Enhanced posture determination for display devices

Abstract

To determine a user's head pose, a head mounted display system having an imaging device may obtain a current image of a real-world environment, points of which correspond to salient points to be used to determine the head pose. The salient points are image patch based and include: a first salient point projected onto the current image from a previous image, and a second salient point extracted from the current image and included in the current image. Each salient point is then matched to a real-world point based on descriptor-based map information indicating the location of the salient point in the real-world environment. Based on the match and based on the relative position of the salient point in the view captured in the current image, an orientation of the imaging device is determined. The orientation may be used to infer the head pose of the wearer of the head mounted display system.

Description

[0001] Incorporated by Reference

[0002] This application claims priority to U.S. Provisional Application No. 62 / 599,620 filed on December 15, 2017 and U.S. Provisional Application No. 62 / 623,606 filed on January 30, 2018. Each of these applications is incorporated herein by reference in its entirety.

[0003] This application further incorporates by reference the entirety of each of the following patent applications: U.S. Application No. 14 / 555,585, filed on November 27, 2014; published on July 23, 2015 as U.S. Publication No. 2015 / 0205126; U.S. Application No. 14 / 690,401, filed on April 18, 2015, which was published on October 22, 2015 as U.S. Publication No. 2015 / 0302652; U.S. Application No. 14 / 212,961, filed on March 14, 2014, which is now published as U.S. Patent No. 2015 / 08 / 16076 issued on August 16, 2016; No. 9,417,452, filed on July 14, 2014; U.S. Application No. 14 / 331,218, filed on July 14, 2014, which was published as U.S. Publication No. 2015 / 0309263 on October 29, 2015; U.S. Application No. 14 / 205,126, filed on March 11, 2014, which was published as U.S. Publication No. 2014 / 0306866 on October 16, 2014; U.S. Application No. 15 / 597,694, filed on May 17, 2017; and U.S. Application No. 15 / 717747, filed on September 27, 2017. Technical Field

[0004] The present disclosure relates to display systems, and more particularly, to augmented reality display systems. Background Art

[0005] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that appears to be real or can be perceived as real. Virtual reality (or "VR") scenes generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality (or "AR") scenes generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user. A mixed reality (or "MR") scene is an AR scene and generally involves virtual objects that are integrated into and responsive to the natural world. For example, in an MR scene, AR image content can be blocked by objects in the real world or perceived as interacting with the objects.

[0006] refer to Figure 1 , shows an augmented scene 10 in which a user of AR technology sees a real-world park-like setting 20 featuring people, trees, buildings in the background, and a concrete platform 30. In addition to these items, the user of AR technology also perceives that he "sees" "virtual content" such as a robotic statue 40 standing on the real-world platform 30 and a cartoon-like avatar character 50 flying by that appears to be an avatar of a bumblebee, even though these elements 40, 50 do not exist in the real world. Because the human visual perception system is complex, it is challenging to produce AR technology that facilitates comfortable, natural, rich presentation of virtual image elements along with other virtual or real-world image elements.

[0007] The systems and methods disclosed herein address various challenges associated with AR and VR technologies. Summary of the invention

[0008] Some non-limiting embodiments include a system comprising one or more imaging devices, one or more processors, and one or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include: obtaining a current image of a real-world environment via the one or more imaging devices, the current image comprising a plurality of points for determining a pose; projecting a first salient point based on an image patch from a previous image onto a corresponding point in a plurality of points in the current image; extracting a second salient point from the current image; providing a corresponding descriptor for the salient point; matching the salient point associated with the current image with a real-world location specified in a descriptor-based map of the real-world environment; and determining a pose associated with the system based on the match, the pose indicating at least an orientation of the one or more imaging devices in the real-world environment.

[0009] In the above embodiment, the operation may further include adjusting the position of the first salient point based on the image block on the current image, wherein adjusting the position includes: obtaining a first image block associated with the first salient point, the first image block including a portion of a previous image containing the first salient point and an area of ​​the previous image surrounding the first salient point; and locating a second image block similar to the first image block in the current image, wherein the first salient point is located at a similar position within the second image block as the first image block. Locating the second image block may include minimizing the difference between the first image block in the previous image and the second image block in the current image. Projecting the first salient point based on the image block onto the current image may be based at least in part on information from an inertial measurement unit of the system. Extracting the second salient point may include: determining that an image region of the current image has a number of salient points projected from the previous image that is less than a threshold; and extracting one or more descriptor-based salient points from the image region, the extracted salient points including the second salient point. The image region may include the entirety of the current image, or the image includes a subset of the current image. The image region may include a subset of the current image, and the system may be configured to adjust a size associated with the subset based on one or more processing constraints or a difference between one or more previously determined postures. Matching a salient point associated with the current image with a real-world location specified in a map of the real-world environment may include: accessing map information, the map information including the real-world location of the salient point and an associated descriptor; and matching the descriptor for the salient point in the current image with the descriptor for the salient point at the real-world location. The operation may further include: projecting the salient point provided in the map information onto the current image, wherein the projection is based on one or more of an inertial measurement unit, an extended Kalman filter, or a visual inertial odometry. The system may be configured to generate a map using at least one or more imaging devices. Determining a pose may be based on the real-world location of the salient point and the relative position of the salient point in a view captured in the current image. The operation may further include: generating an image patch associated with a corresponding salient point extracted from the current image, so that for a subsequent image of the current image, the image patch includes the salient point that can be used to be projected onto the subsequent image. Providing the descriptor may include generating a descriptor for each of the salient points.

[0010] In other embodiments, an augmented reality display system is provided. The augmented reality display device includes one or more imaging devices and one or more processors. The processor is configured to obtain a current image of a real-world environment; perform frame-to-frame tracking on the current image so that image-block-based salient points included in a previous image are projected onto the current image; perform map-to-frame tracking on the current image so that descriptor-based salient points included in a map database are matched with salient points in the current image; and determine a posture associated with the display device.

[0011] In the above embodiments, frame-to-frame tracking may further include using photometric error optimization to refine the position of the projected image block. Map-to-frame tracking may further include determining descriptors for salient points based on the image block, and matching the descriptors for the salient points with descriptor-based salient points in the map database. The one or more processors may be further configured to generate the map database using at least one or more imaging devices. The augmented reality display system may further include a plurality of waveguides configured to output light having different wavefront divergences corresponding to different depth planes, wherein the output light is positioned based at least in part on a posture associated with the display device.

[0012] In other embodiments, a method is provided. The method includes: obtaining a current image of a real-world environment via one or more imaging devices, the current image including a plurality of points for determining a pose; projecting first image patch-based salient points from a previous image onto corresponding points of a plurality of points in the current image; extracting second salient points from the current image; providing corresponding descriptors for the salient points; matching the salient points associated with the current image with real-world locations specified in a descriptor-based map of the real-world environment; and determining a pose associated with a display device based on the match, the pose indicating at least an orientation of the one or more imaging devices in the real-world environment.

[0013] In these embodiments, the method may further include: adjusting the position of the first salient point based on the image patch on the current image, wherein adjusting the position includes: obtaining a first image patch associated with the first salient point, the first image patch including a portion of a previous image containing the first salient point and an area of ​​the previous image surrounding the first salient point; and locating a second image patch similar to the first image patch in the current image, wherein the first salient point is located in a similar position within the second image patch as the first image patch. Locating the second image patch may include minimizing a difference between the first image patch in the previous image and the second image patch in the current image. Projecting the first salient point based on the image patch onto the current image may be based at least in part on information from an inertial measurement unit of the system. Extracting the second salient point may include: determining that an image region of the current image has less than a threshold number of salient points projected from the previous image; and extracting one or more descriptor-based salient points from the image region, the extracted salient points including the second salient point. The image region may include the entirety of the current image, or the image includes a subset of the current image. The image region may include a subset of the current image, and the system may be configured to adjust a size associated with the subset based on one or more processing constraints or a difference between one or more previously determined poses. Matching salient points associated with the current image with real-world locations specified in a map of the real-world environment may include: accessing map information, the map information including real-world locations of salient points and associated descriptors; and matching descriptors for salient points in the current image with descriptors for salient points at real-world locations. The method may further include: projecting salient points provided in the map information onto the current image, wherein the projection is based on one or more of an inertial measurement unit, an extended Kalman filter, or a visual inertial odometry. The system may be configured to generate a map using at least one or more imaging devices. Determining a pose may be based on the real-world location of the salient point and a relative position of the salient point in a view captured in the current image. The method may further include: generating an image patch associated with a corresponding salient point extracted from the current image, so that for a subsequent image of the current image, the image patch includes a salient point that can be used to be projected onto the subsequent image. Providing descriptors may include generating a descriptor for each of the salient points. The method may further include generating a map using at least one or more imaging devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A user's view of augmented reality (AR) through an AR device is shown.

[0015] Figure 2 A conventional display system for simulating a three-dimensional image for a user is shown.

[0016] Figures 3A-3CThe relationship between the radius of curvature and the focus radius is shown.

[0017] Figure 4A A representation of the accommodation-vergence response of the human visual system is shown.

[0018] Figure 4B Examples of different accommodation and vergence states for a pair of eyes of a user are shown.

[0019] Figure 4C An example of a representation of an overhead view of a user viewing content via a display system is shown.

[0020] Figure 4D Another example of a representation of an overhead view of a user viewing content via a display system is shown.

[0021] Figure 5 Aspects of a method for simulating three-dimensional images by modifying wavefront divergence are shown.

[0022] Figure 6 An example of a waveguide stack for outputting image information to a user is shown.

[0023] Figure 7 An example of an outgoing light beam output by a waveguide is shown.

[0024] Figure 8 An example of a stacked waveguide assembly is shown where each depth plane includes an image formed using multiple different assembly colors.

[0025] Fig. 9A A cross-sectional side view of an example of a group of stacked waveguides is shown, wherein each stacked waveguide includes an in-coupling optical element.

[0026] Fig. 9B Shows Fig. 9A A perspective view of an example of multiple stacked waveguides.

[0027] Fig. 9C Shows Fig. 9A and 9B A top plan view of an example of multiple stacked waveguides.

[0028] Fig.9D An example of a wearable display system is shown.

[0029] Fig. 10A A flow chart of an example process for determining a posture of a display system and a posture of a user's head is shown.

[0030] Fig. 10B An example image region of the current image is shown.

[0031] Fig.11A flow chart of an example process for frame-to-frame tracking is shown.

[0032] Fig. 12A Examples of previous and current images are shown.

[0033] Fig. 12B Shows the projection to Fig. 12A Examples of image patches on the current image.

[0034] Fig.13 A flow chart showing an example process for map-to-frame tracking is shown.

[0035] Fig.14A Examples of previous and current images are shown.

[0036] Fig. 14B An example of frame-to-frame tracking following map-to-frame tracking is shown.

[0037] Fig.15 A flow chart of an example process for determining head pose is shown. DETAILED DESCRIPTION

[0038] Display systems such as augmented reality (AR) or virtual reality (VR) display systems may present content to a user (or viewer) in different areas of the user's field of view. For example, an augmented reality display system may present virtual content to a user that appears to be placed in a real-world environment to the user. As another example, a virtual reality display system may present content via a display so that the content may appear to be three-dimensional and placed in a three-dimensional environment to the user. The placement of this content, for example relative to the user, may positively or negatively affect the sense of reality associated with the presented content and the user's sense of comfort when wearing the display system. Because the placement of content may depend on the head posture of the user of the display system, as described below, these display systems may be enhanced by utilizing precise schemes to determine the head posture.

[0039] The posture of the user's head can be understood as the orientation of the user's head relative to the real-world environment (e.g., relative to a coordinate system associated with the real-world environment) (e.g., the pitch, yaw and / or rotation of the head). In some embodiments, the display system may also have a posture corresponding to a specific orientation of the display system (e.g., an AR or VR display device) or a portion of the display system relative to the real-world environment (e.g., relative to a coordinate system associated with the real-world environment). The posture may optionally generally represent the orientation relative to the coordinate system in the real-world environment. For example, if the user rotates the display system mounted on his / her head (e.g., by rotating his / her head), the posture of the user's head and the display system may be adjusted according to the rotation. Therefore, the content presented to the user may be adjusted based on the posture of the user's head, which may also change the posture of the display of the display system mounted on the user's head. In some embodiments, the posture of the display system may be determined, and the user's head posture may be inferred from the display system posture. By determining the head posture, when the user moves in the real-world environment, the content may be realistically adjusted in position and orientation based on the determined user head posture. Some examples are described below.

[0040] With respect to augmented reality (AR) and virtual reality (VR) display systems, the sense of reality can be enhanced if a user can move around the presented virtual content, and the presented virtual content may appear to remain substantially in a fixed real-world location. For example, Figure 1 The robot 40 statue shown can be virtual content presented to the user. As the user walks toward or around the robot 40, if the robot 40 appears to remain in the same position in the park, the augmented reality scene 10 will appear more realistic to the user. Therefore, the user can view different perspectives of the robot 40, different parts of the robot 40, and so on. In order to ensure that the robot 40 appears to be a fixed real statue, the display system can utilize a certain posture when rendering the virtual content. For example, the display system can obtain information indicating that the user rotates his / her head at a specific angle. This rotation can inform the placement of the virtual content so that the robot 40 will appear to remain upright like a statue.

[0041] As another example, a user may play a first-person video game while wearing a display system. In this example, the user may quickly lower his / her head or rotate his / her head to avoid the path of a virtual enemy object thrown at the user, as presented to the user by the display system. The motion (e.g., the lowering or rotation of the head) may be tracked, and the user's head pose may be determined. In this way, the display system may determine whether the user successfully avoided the enemy object.

[0042] Systems for determining posture can be complex. Example schemes for determining head posture may utilize sensors and emitters of light. For example, an infrared emitter may emit pulses of infrared light from a fixed position in a real-world environment (e.g., the emitter may be located in a room around the device). A display device worn by a user may include a sensor to detect these pulses. Thus, the display device may determine its orientation relative to the fixed emitter. Similarly, the display device may determine its position in the real-world environment based on the fixed emitter. As another example, the display device may include an emitter of fixed light (e.g., visible light or infrared light), and one or more cameras may be located in the real-world environment that tracks the emission of light. In this example, as the display device rotates, the camera may detect that the emission of light rotates from an initial position. Therefore, these example schemes may require complex hardware to determine the posture of the display device.

[0043] The display systems described herein (e.g., Fig.9D The display system 60 shown in FIG. 60 may determine an accurate pose estimate without the complexity and rigidity of a fixed light emitter. In some embodiments, the display system (e.g., such as a display system worn by a user) may be determined. Fig.9D The display system 70 shown coupled to the frame 80 can be used to determine the posture of the user's head. Based on the display system posture, the user's head posture can be determined. The display system can determine its posture without the user setting up complex hardware in the room, and the user does not need to stay in the room. As will be described, the display system can determine its posture by using an imaging device (e.g., an optical imaging device such as a camera) on the display system. The imaging device can obtain images of the real-world environment, and based on these images, the display system can determine its posture. Through the techniques described herein, the display system can advantageously determine the posture while limiting processing and storage requirements. In this way, display systems with limited memory and computing budgets such as AR and MR display systems can effectively determine posture estimates and increase realism for users.

[0044] To determine pose, the display system may utilize (1) image patch-based tracking of distinguishable points (e.g., different isolated portions of an image) between successive images of an environment captured by the display system (referred to herein as "frame-to-frame tracking"), and (2) matching points of interest in a current image with a descriptor-based map of known real-world locations of corresponding points of interest (referred to herein as "map-to-frame tracking"). In frame-to-frame tracking, the display system may track specific points of interest (referred to herein as "salient points"), such as corner points, between captured images of the real-world environment. For example, the display system may identify the location of a visual point of interest in a current image that is included in (e.g., located in) a previous image. This identification may be accomplished using, for example, a photometric error minimization method. In map-to-frame tracking, the display system may access map information that indicates the real-world locations (e.g., three-dimensional coordinates) of the points of interest, and match the points of interest included in the current image with the points of interest indicated in the map information. Information about the points of interest may be stored as descriptors, such as in a map database. The display system may then compute its pose based on the matched visual features. The following will refer to Fig. 10A Generating map information is described in more detail.As used herein, in some embodiments, a point may refer to a discrete pixel of an image or a group of pixels corresponding to a region of an image.

[0045] As described above, to determine a gesture, the display system may utilize distinguishable visual features, referred to herein as "salient points." In this specification, a salient point corresponds to any unique portion of a real-world environment that can be tracked. For example, a salient point may be a corner point. A corner point may represent a substantially perpendicular intersection of two lines, and may include a scratch on a table, a mark on a wall, the number '7' on a keyboard, and the like. As will be described, corner points may be detected from an image obtained by an imaging device according to a corner point detection scheme. Example corner point detection schemes may include Harris corner point detection, accelerated segmentation test feature (FAST) corner point detection, and the like.

[0046] With respect to frame-to-frame tracking, the display system may track salient points from a previous image to a current image by projecting each tracked salient point from the previous image onto the current image. For example, the display system may utilize trajectory prediction or, alternatively, information from an inertial measurement unit to determine an optical flow between the current image and the previous image. The optical flow may represent the movement of the user from the time the previous image was obtained to the time the current image was obtained. The trajectory prediction may tell the location in the current image to which the salient points included in the previous image correspond. The display system may then obtain an image portion (referred to herein as an "image patch") surrounding each salient point in the previous image and determine a matching image portion in the current image. An image patch may be, for example, an MxN pixel region surrounding each salient point in the previous image, where M and N are positive integers. In order to match an image patch from a previous image to a current image, the display system may identify an image patch in the current image that has a reduced (e.g., minimized) photometric error between the image patch and the image patch of the previous image. A salient point may be understood as a specific, consistent two-dimensional image location located within an image patch. For example, the centroid of the matching image patch in the current image may correspond to the tracked salient point. Thus, the projection from the previous image onto the current image approximately locates the salient point and the associated image patch within the current image, and the location may be determined, for example, using photometric error minimization to refine the location that minimizes the difference in pixel intensity between the image patch and a particular region of the current image.

[0047] With respect to map-to-frame tracking, the display system may extract salient points from the current image (e.g., identify locations in the current image that correspond to new salient points). For example, the display may extract salient points from an image region in the current image that has less than a threshold number of tracked salient points (e.g., determined from frame-to-frame tracking). The display system may then match the salient points in the current image (e.g., newly extracted salient points, tracked salient points) to corresponding real-world locations based on descriptor-based map information. As described herein, the display system may generate a descriptor for each salient point that uniquely describes an attribute (e.g., visual) of the salient point. The map information may similarly store descriptors about real-world salient points. Based on the matched descriptors, the display system may determine the real-world location of the salient point included in the current image. Thus, the display system may determine its orientation relative to the real-world location and determine its pose, which may then be used to determine the head pose.

[0048] It will be appreciated that highly accurate tracking of salient points between images may be achieved using a photometric error minimization scheme, such as by comparison of image patches as described above. In fact, sub-pixel accuracy in tracking salient points between a previous image and a current image may be achieved. Relatively speaking, descriptors may be less accurate in tracking salient points between images, but use less storage than image patches used for photometric error minimization. Because descriptors may be less accurate in tracking salient points between images, the determined pose estimate may be larger than using photometric error minimization. Although accurate, using image patches may require storing an image patch for each salient point. Because descriptors may be alphanumeric values ​​that describe visual characteristics of a salient point and / or an image region (e.g., a histogram) surrounding the salient point, descriptors may be one or more orders of magnitude smaller than image patches.

[0049] Thus, as described herein, a display device may take advantage of image patch-based photometric error minimization and descriptors to allow a robust and memory-efficient posture determination process. For example, frame-to-frame tracking may utilize image patch-based photometric error minimization to accurately track salient points between images. In this manner, salient points may be tracked with, for example, sub-pixel accuracy. However, over time (e.g., across multiple frames or images), small errors may be introduced such that drift caused by accumulated errors in tracking salient points may become apparent over a threshold number of images. Such drift may reduce the accuracy of posture determination. Therefore, in some embodiments, map-to-frame tracking may be utilized to link each salient point to a real-world location. For example, in map-to-frame tracking, salient points are matched with salient points stored in map information. Thus, the real-world coordinates of each salient point may be identified.

[0050] If photometric error minimization is used for map-to-frame tracking, the map information will store an image patch for each salient point identified in the real-world environment. Since there may be thousands, hundreds of thousands, etc. of salient points indicated in the map information, the storage requirements will be large. Advantageously, the use of descriptors can reduce the storage requirements associated with map-to-frame tracking. For example, the map information can store the real-world coordinates of each salient point and a descriptor for the salient point. In some embodiments, since the size of the descriptor can be at least an order of magnitude smaller than the image patch, the map information can be greatly reduced.

[0051] As will be described below, the display system may therefore utilize tile-based frame-to-frame tracking and descriptor-based map-to-frame tracking. For example, the display system may track salient points between consecutive images obtained in a real-world environment. As described above, tracking salient points may include projecting salient points from a previous image onto a current image. By using tile-based photometric error minimization, the positions of the tracked salient points may be determined very accurately in the current image. The display system may then identify image regions of the current image that include tracked salient points that are less than a threshold measure. For example, the current image may be divided into different image regions, each of which is 1 / 4, 1 / 8, 1 / 16, 1 / 32, a user-selectable size, etc., of the current image. As another example, the display system may analyze the sparsity of the current image with respect to the tracked salient points. In this example, the display system may determine whether any region of the image (e.g., a region of a threshold size) includes less than a threshold number of salient points or less than a threshold density of salient points. Alternatively, the image region may be the entire current image, such that the display system may identify whether the entirety of the current image includes tracked salient points that are less than a threshold measurement value. The display system may then extract new salient points from the identified image region and generate descriptors for each salient point of the current image (e.g., tracked salient points and newly extracted salient points). By matching each generated descriptor with the descriptors of the salient points indicated in the map information, the real-world location of each salient point in the current image may be identified. Thus, the posture of the display system may be determined. Subsequently, as described herein, salient points included in the current image may be tracked in subsequent images.

[0052] Since new salient points can be extracted only in image regions with tracked salient points less than a threshold measurement value, salient point tracking can potentially utilize a large number of identical tracked salient points between consecutive image frames. As described above, tracking can be performed by photometric error minimization, thereby ensuring highly accurate positioning of salient points between images. In addition, since these identical tracked salient points will be matched with map information in consecutive image frames, jitter in posture determination can be reduced. In addition, processing requirements can be reduced because the display system may only be required to extract new salient points in specific image regions. In addition, since the salient points in the current image are matched with the map information, drift in posture determination can be reduced. Optionally, some current images may not require map-to-frame tracking. For example, the user may be viewing in a substantially similar real-world area, so that the display system can maintain a similar posture. In this example, the current image may not include image regions with tracked salient points less than a threshold measurement value. Therefore, frame-to-frame tracking can be used only to determine the posture of the display system. Optionally, map-to-frame tracking can be utilized even if no image region includes less than a threshold number of tracked salient points. For example, descriptors may be generated for tracked salient points and compared to map information without extracting new salient points. In this way, the display system may perform less processing, thereby saving processing resources and reducing energy consumption.

[0053] Reference will now be made to the drawings, wherein like reference numerals refer to like parts throughout. Unless otherwise specifically noted, the drawings are schematic and are not necessarily drawn to scale.

[0054] Figure 2 A conventional display system for simulating a three-dimensional image for a user is shown. It should be understood that the user's eyes are separated, and when viewing a real object in space, each eye will have a slightly different view of the object, and an image of the object may be formed at different locations on the retina of each eye. This may be referred to as binocular parallax, and may be utilized by the human visual system to provide depth perception. Conventional display systems simulate binocular parallax by presenting two different images 190, 200-one for each eye 210, 220-with slightly different views of the same virtual object, the different views corresponding to the view of the virtual object that each eye will see, the virtual object being a virtual object of the real object located at the desired depth. These images provide binocular cues, which the user's visual system may interpret to obtain depth perception.

[0055] Continue to refer Figure 2, the images 190, 200 are spaced a distance 230 from the eyes 210, 220 on the z-axis. The z-axis is parallel to the optical axis of the viewer, whose eyes are fixed on an object at optical infinity directly in front of the viewer. The images 190, 200 are flat and located at a fixed distance from the eyes 210, 220. Based on the slightly different views of the virtual objects in the images presented to the eyes 210, 220, respectively, the eyes can naturally rotate so that the image of the object falls on a corresponding point on the retina of each eye to maintain a single binocular vision. This rotation can cause the line of sight of each eye 210, 220 to converge to a point in space where the virtual object is perceived to exist. As a result, providing a three-dimensional image generally involves providing binocular cues that can manipulate the convergence of the user's eyes 210, 220, and the human visual system interprets it as providing depth perception.

[0056] However, producing realistic and comfortable depth perception is challenging. It should be understood that light from objects at different distances from the eye has wavefronts with different amounts of divergence. Figures 3A-3C The relationship between distance and light divergence is shown. The distance between the object and the eye 210 is represented by the order of decreasing distance R1, R2 and R3. Figures 3A-3C As shown, as the distance to the object decreases, the light becomes more divergent. Conversely, as the distance increases, the light becomes more collimated. In other words, it can be said that the light field generated by a point (object or part of an object) has a spherical wavefront curvature that is a function of how far the point is from the user's eye. As the distance between the object and the eye 210 decreases, the curvature increases. Although for clarity of illustration in Figures 3A-3C Only a single eye 210 is shown in and other figures herein, but it should be understood that the discussion regarding eye 210 is applicable to both eyes 210 and 220 of a viewer.

[0057] Continue to refer Figures 3A-3C, light from an object being gazed at by the viewer's eye may have different degrees of wavefront divergence. Due to different amounts of wavefront divergence, light may be focused differently by the lens of the eye, which in turn may require the lens to assume different shapes to form a focused image on the retina of the eye. In the absence of a focused image formed on the retina, the resulting retinal blur acts as a cue for accommodation, which causes a change in the shape of the lens of the eye until a focused image is formed on the retina. For example, a cue about accommodation may trigger the ciliary muscles around the lens of the eye to relax or contract, thereby adjusting the force applied to the suspensory ligaments that hold the lens, thereby causing the shape of the lens of the eye to change until the retinal blur of the object being gazed is eliminated or minimized, thereby forming a focused image of the object being gazed on the retina of the eye (e.g., the fovea). The process by which the lens of the eye changes shape may be referred to as accommodation, and the shape of the lens of the eye required to form a focused image of the object being gazed on the retina of the eye (e.g., the fovea) may be referred to as an accommodation state.

[0058] Reference now Figure 4A , shows a representation of the accommodation-vergence response of the human visual system. Movement of the eyes to fixate on an object causes the eyes to receive light from the object, where the light forms an image on each of the retinas of the eyes. The presence of retinal blur in the image formed on the retina can provide a cue as to accommodation, and the relative position of the images on the retina can provide a cue as to vergence. Cues as to accommodation cause accommodation to occur, resulting in the lenses of the eyes each assuming a specific accommodation state that forms a focused image of the object on the retina of the eye (e.g., the fovea). On the other hand, cues as to vergence cause vergence movements (rotations of the eyes) to occur, such that the images formed on each retina of each eye are at corresponding retinal points that maintain single binocular vision. At these positions, the eyes can be said to have been in a specific vergence state. Continue with reference to Figure 4A , accommodation can be understood as the process by which the eye achieves a specific accommodation state, and vergence can be understood as the process by which the eye achieves a specific vergence state. Figure 4A As shown in , if the user looks at another object, the accommodation and convergence states of the eyes can change. For example, if the user looks at a new object at a different depth on the z-axis, the accommodation state can change.

[0059] Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of vergence and accommodation. As described above, the vergence movement of the two eyes relative to each other (e.g., rotation of the eyes so that the pupils move toward or away from each other to converge the eyes' line of sight to focus on an object) is closely related to the accommodation of the lens of the eye. Under normal circumstances, changing the shape of the lens of the eye to change focus from one object to another object located at a different distance will automatically result in a matching change in vergence to the same distance under a relationship known as the "accommodation-vergence reflex." Likewise, under normal circumstances, a change in vergence will induce a matching change in the shape of the lens.

[0060] Reference now Figure 4B , examples of different accommodation and vergence states of eyes are shown. Eye pair 222a is fixated on an object at optical infinity, while eye pair 222b is fixated on object 221 that is less than optical infinity. Notably, the vergence state of each pair of eyes is different, with eye pair 222a pointing straight ahead and eye pair 222 converging on object 221. The accommodation state of the eyes forming each eye pair 222a and 222b may also be different, as indicated by the different shapes of the lenses 210a, 220a.

[0061] Undesirably, many users of conventional "3-D" display systems find these conventional systems uncomfortable or simply do not perceive a sense of depth due to the mismatch between the accommodation and vergence states in these displays. As described above, many stereoscopic or "3-D" display systems display a scene by providing a slightly different image to each eye. Such systems are uncomfortable for many viewers because they, among other things, only provide a different presentation of the scene and cause a change in the vergence state of the eyes, but without a corresponding change in the accommodation state of those eyes. However, the image is shown by a display at a fixed distance from the eye, so that the eye views all image information in a single accommodation state. This arrangement counteracts the "accommodation-vergence reflex" by causing a change in the vergence state without a matching change in the accommodation state. This mismatch is believed to cause discomfort to the viewer. A display system that provides a better match between accommodation and vergence can form a more realistic and comfortable three-dimensional image simulation.

[0062] Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Therefore, by providing the eye with different representations of images corresponding to each of these limited number of depth planes, a highly convincing simulation of perceived depth can be achieved. In some embodiments, the different representations can provide cues about vergence and matching cues about accommodation, thereby providing a physiologically correct accommodation-vergence match.

[0063] Continue to refer Figure 4B , two depth planes 240 are shown, corresponding to different distances in space from the eyes 210, 220. For a given depth plane 240, vergence cues may be provided by displaying appropriately different perspective images for each eye 210, 220. In addition, for a given depth plane 240, the light forming the image provided to each eye 210, 220 may have a wavefront divergence corresponding to the light field produced by a point at the distance of that depth plane 240.

[0064] In the illustrated embodiment, the distance along the z-axis of the depth plane 240 containing point 221 is 1 m. As used herein, the distance or depth along the z-axis may be measured using a zero point located at the exit pupil of the user's eye. Thus, on the optical axis of those eyes where the eyes are pointed toward optical infinity, the depth plane 240 located at a depth of 1 m corresponds to a distance of 1 m from the exit pupil of the user's eye. As an approximation, the depth or distance along the z-axis may be measured from a display in front of the user's eye (e.g., from the surface of a waveguide), plus the value of the distance between the device and the exit pupil of the user's eye. This value may be referred to as eye relief and corresponds to the distance between the exit pupil of the user's eye and the display that the user wears in front of the eye. In practice, the value of eye relief may be a standardized value that is commonly used for all viewers. For example, it may be assumed that the eye relief is 20 mm, and a depth plane with a depth of 1 m may be at a distance of 980 mm in front of the display.

[0065] Reference now Figure 4C and 4D , respectively, show examples of matched accommodation-vergence distances and mismatched accommodation-vergence distances. Figure 4C As shown, the display system may provide an image of a virtual object to each eye 210, 220. The image may cause the eyes 210, 220 to present a vergence state in which the eyes converge on a point 15 on a depth plane 240. In addition, the image may be formed by light having a wavefront curvature corresponding to a real object at the depth plane 240. As a result, the eyes 210, 220 present an accommodation state in which the image is focused on the retinas of those eyes. Therefore, the user may perceive the virtual object at a point 15 on the depth plane 240.

[0066] It should be understood that each of the accommodation and vergence states of the eyes 210, 220 is associated with a particular distance on the z-axis. For example, an object at a particular distance from the eyes 210, 220 causes those eyes to assume a particular accommodation state based on the distance of the object. The distance associated with a particular accommodation state may be referred to as the accommodation distance A. d Similarly, there is a specific vergence distance V associated with the eyes in a specific vergence state or position relative to each other. dIn the case where the accommodation distance and the vergence distance match, the relationship between accommodation and vergence can be said to be physiologically correct. This is considered to be the most comfortable scenario for the viewer.

[0067] However, in stereoscopic displays, the accommodation distance and the vergence distance may not always match. Figure 4D As shown, the image displayed to the eyes 210, 220 can be displayed with a wavefront divergence corresponding to the depth plane 240, and the eyes 210, 220 can present a specific accommodation state in which points 15a, 15b on the depth plane are focused. However, the image displayed to the eyes 210, 220 can provide a prompt about the vergence in which the eyes 210, 220 are converged on the point 15 that is not located on the depth plane 240. As a result, in some embodiments, the accommodation distance corresponds to the distance from the exit pupil of the eyes 210, 220 to the depth plane 240, while the vergence distance corresponds to the larger distance from the exit pupil of the eyes 210, 220 to the point 15. The accommodation distance is different from the vergence distance. Therefore, there is an accommodation-vergence mismatch. This mismatch is considered to be unwanted and may cause discomfort to the user. It should be understood that the mismatch corresponds to the distance (e.g., V d -A d ) and can be characterized using diopters.

[0068] In some embodiments, it should be appreciated that reference points other than the exit pupils of eyes 210, 220 may be used to determine the distance used to determine the accommodation-vergence mismatch, as long as the same reference point is used for the accommodation distance and the vergence distance. For example, the distance may be measured from the cornea to the depth plane, from the retina to the depth plane, from the eyepiece (e.g., a waveguide of a display device) to the depth plane, etc.

[0069] Without being limited by theory, it is believed that the user may still perceive accommodation-vergence mismatches up to about 0.25 diopters, up to about 0.33 diopters, and up to about 0.5 diopters as physiologically correct, without the mismatch itself causing significant discomfort. In some embodiments, the display systems disclosed herein (e.g., display system 250, Figure 6 ) presents an image to a viewer with an accommodation-vergence mismatch of about 0.5 diopters or less. In some other embodiments, the accommodation-vergence mismatch of the image provided by the display system is about 0.33 diopters or less. In other embodiments, the accommodation-vergence mismatch of the image provided by the display system is about 0.25 diopters or less, including about 0.1 diopters or less.

[0070] Figure 5Aspects of a method for simulating a three-dimensional image by modifying wavefront divergence are shown. The display system includes a waveguide 270 configured to receive light 770 encoded with image information and output the light to an eye 210 of a user. The waveguide 270 may output light 650 having a defined amount of wavefront divergence corresponding to the wavefront divergence of a light field generated by a point on a desired depth plane 240. In some embodiments, the same amount of wavefront divergence is provided for all objects presented on the depth plane. Additionally, it will be described that image information from a similar waveguide may be provided to the user's other eye.

[0071] In some embodiments, a single waveguide may be configured to output light with a set amount of wavefront divergence corresponding to a single or limited number of depth planes and / or the waveguide may be configured to output light of a limited wavelength range. Thus, in some embodiments, multiple waveguides or waveguide stacks may be utilized to provide different amounts of wavefront divergence for different depth planes and / or output light of different wavelength ranges. As used herein, it will be understood that the contour of a flat or curved surface may be followed at a depth plane. In some embodiments, advantageously for simplicity, a depth plane may follow the contour of a flat surface.

[0072] Figure 6 An example of a waveguide stack for outputting image information to a user is shown. The display system 250 includes a waveguide stack or stacked waveguide assembly 260 that can be used to provide a three-dimensional perception to the eye / brain using multiple waveguides 270, 280, 290, 300, 310. It can be understood that in some embodiments, the display system 250 can be considered a light field display. In addition, the waveguide assembly 260 can also be referred to as an eyepiece.

[0073] In some embodiments, the display system 250 may be configured to provide a substantially continuous cue about vergence and a plurality of discrete cues about accommodation. The cues about vergence may be provided by displaying a different image to each eye of the user, and the cues about accommodation may be provided by outputting light forming the image with a selectable discrete amount of wavefront divergence. In other words, the display system 250 may be configured to output light with variable levels of wavefront divergence. In some embodiments, each discrete level of wavefront divergence corresponds to a particular depth plane and may be provided by a particular one of the waveguides 270, 280, 290, 300, 310.

[0074] Continue to refer Figure 6, the waveguide assembly 260 may also include a plurality of features 320, 330, 340, 350 located between the waveguides. In some embodiments, the features 320, 330, 340, 350 may be one or more lenses. The waveguides 270, 280, 290, 300, 310 and / or the plurality of lenses 320, 330, 340, 350 may be configured to send image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to the depth plane. The image injection device 360, 370, 380, 390, 400 may serve as a light source for the waveguide and may be used to inject image information into the waveguides 270, 280, 290, 300, 310, each of which may be configured to distribute incident light through each corresponding waveguide for output to the eye 210 as described herein. Light exits the output surfaces 410 , 420 , 430 , 440 , 450 of the image injection devices 360 , 370 , 380 , 390 , 400 and is injected into the corresponding input surfaces 460 , 470 , 480 , 490 , 500 of the waveguides 270 , 280 , 290 , 300 , 310 . In some embodiments, each of the input surfaces 460, 470, 480, 490, 500 may be an edge of a respective waveguide, or may be a portion of a major surface of a respective waveguide (i.e., one of the waveguide surfaces that directly faces the world 510 or the viewer's eye 210). In some embodiments, a single light beam (e.g., a collimated light beam) may be injected into each waveguide so as to output an entire field of view of a cloned collimated light beam directed toward the eye 210 at a particular angle (and divergence) corresponding to a depth plane associated with the particular waveguide. In some embodiments, a single one of the image injection devices 360, 370, 380, 390, 400 may be associated with and inject light into multiple (e.g., three) of the waveguides 270, 280, 290, 300, 310.

[0075] In some embodiments, the image injection devices 360, 370, 380, 390, 400 are separate displays, each of which generates image information for injection into a respective waveguide 270, 280, 290, 300, 310. In some other embodiments, the image injection devices 360, 370, 380, 390, 400 are outputs of a single multiplexed display, which may pipe image information to each of the image injection devices 360, 370, 380, 390, 400, for example, via one or more light conduits (such as fiber optic cables). It will be appreciated that the image information provided by the image injection devices 360, 370, 380, 390, 400 may include light of different wavelengths or colors (e.g., different component colors as discussed herein).

[0076] In some embodiments, the light injected into the waveguides 270, 280, 290, 300, 310 is provided by a light projector system 520 including a light module 530, which may include a light emitter such as a light emitting diode (LED). The light from the light module 530 may be directed to and modified by a light modulator 540 (e.g., a spatial light modulator) via a beam splitter 550. The light modulator 540 may be configured to change the perceived intensity of the light injected into the waveguides 270, 280, 290, 300, 310 to decode the light having image information. Examples of spatial light modulators include liquid crystal displays (LCDs), including liquid crystal on silicon (LCOS) displays. It should be understood that the image injection devices 360, 370, 380, 390, 400 are shown schematically, and in some embodiments, these image injection devices may represent different light paths and locations in a common projection system that is configured to output light into associated waveguides in the waveguides 270, 280, 290, 300, 310. In some embodiments, the waveguides of the waveguide assembly 260 may act as ideal lenses while relaying the light injected into the waveguides out to the user's eyes. In this concept, the object may be the spatial light modulator 540, and the image may be an image on a depth plane.

[0077] In some embodiments, the display system 250 may be a scanning fiber display including one or more scanning fibers configured to project light into one or more waveguides 270, 280, 290, 300, 310 and ultimately to the viewer's eye 210 in various patterns (e.g., raster scan, spiral scan, Lissajous pattern, etc.). In some embodiments, the image injection devices 360, 370, 380, 390, 400 shown may schematically represent a single scanning fiber or a scanning fiber bundle configured to inject light into one or more of the waveguides 270, 280, 290, 300. In some other embodiments, the image injection devices 360, 370, 380, 390, 400 shown may schematically represent a plurality of scanning fibers or a plurality of scanning fiber bundles, each of which is configured to inject light into an associated one of the waveguides 270, 280, 290, 300, 310. It should be understood that one or more optical fibers may be configured to transmit light from the optical module 530 to one or more waveguides 270, 280, 290, 300, 310. It should be understood that one or more intervening optical structures may be provided between the scanning optical fiber or optical fibers and the one or more waveguides 270, 280, 290, 300, 310 to, for example, redirect light exiting the scanning optical fiber into the one or more waveguides 270, 280, 290, 300, 310.

[0078] Controller 560 controls the operation of one or more of stacked waveguide assemblies 260, including the operation of image injection devices 360, 370, 380, 390, 400, light source 530, and light modulator 540. In some embodiments, controller 560 is part of local data processing module 140. Controller 560 includes programming (e.g., instructions in a non-transitory medium) that regulates the timing and provision of image information to waveguides 270, 280, 290, 300, 310, for example, according to any of the various schemes disclosed herein. In some embodiments, controller 560 can be a single integral device, or a distributed system connected by wired or wireless communication channels. In some embodiments, controller 560 can be a processing module 140 or 150 ( Fig.9D ) part.

[0079] Continue to refer Figure 6, the waveguides 270, 280, 290, 300, 310 may be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 270, 280, 290, 300, 310 may each be planar or have other shapes (e.g., curved) having a top major surface and a bottom major surface and an edge extending between these top major surfaces and the bottom major surface. In the illustrated configuration, the waveguides 270, 280, 290, 300, 310 may each include an outcoupling optical element 570, 580, 590, 600, 610 configured to extract light out of the waveguide by redirecting the light propagating within each respective waveguide out of the waveguide to output image information to the eye 210. The extracted light may also be referred to as outcoupled light, and the outcoupling optical element light may also be referred to as a light extraction optical element. The extracted light beam may be output by the waveguide at a location where the light propagating in the waveguide impinges on the light extraction optical element. The outcoupling optical element 570, 580, 590, 600, 610 may, for example, include a grating of diffractive optical features, as discussed further herein. Although illustrated as being disposed at the bottom major surface of the waveguide 270, 280, 290, 300, 310 for ease of description and clarity of drawing, in some embodiments, the outcoupling optical element 570, 580, 590, 600, 610 may be disposed at the top and / or bottom major surface and / or may be disposed directly in the volume of the waveguide 270, 280, 290, 300, 310, as discussed further herein. In some embodiments, the outcoupling optical element 570, 580, 590, 600, 610 may be formed in a material layer attached to a transparent substrate to form the waveguide 270, 280, 290, 300, 310. In some other embodiments, the waveguides 270, 280, 290, 300, 310 may be a single piece of material and the outcoupling optical elements 570, 580, 590, 600, 610 may be formed on a surface of and / or within the interior of the piece of material.

[0080] Continue to refer Figure 6As discussed herein, each waveguide 270, 280, 290, 300, 310 is configured to output light to form an image corresponding to a particular depth plane. For example, the waveguide 270 closest to the eye may be configured to transmit collimated light, as injected into such waveguide 270, to the eye 210. The collimated light may represent an optical infinity focal plane. The next up-going waveguide 280 may be configured to send the collimated light out through a first lens 350 (e.g., a negative lens) before it can reach the eye 210; such a first lens 350 may be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next up-going waveguide 280 as coming from a first focal plane that is closer inward from optical infinity toward the eye 210. Similarly, the third upstream waveguide 290 causes its output light to pass through the first lens 350 and the second lens 340 before reaching the eye 210; the combined optical power of the first lens 350 and the second lens 340 can be configured to produce another incremental wavefront curvature so that the eye / brain interprets the light from the third waveguide 290 as coming from a second focal plane that is closer inward toward the person from optical infinity than the light from the next upstream waveguide 280.

[0081] The other waveguide layers 300, 310 and lenses 330, 320 are similarly configured, with the highest waveguide 310 in the stack sending its output through all the lenses between it and the eye for an aggregate focal power representing the focal plane closest to the person. To compensate for the stack of lenses 320, 330, 340, 350 when viewing / interpreting light from the world 144 on the other side of the stacked waveguide assembly 178, a compensating lens layer 620 may be provided at the top of the stack to compensate for the aggregate focal power of the underlying lens stack 320, 330, 340, 350. This configuration provides as many perceived focal planes as there are available waveguide / lens pairings. The out-coupling optics of the waveguide and the focusing aspects of the lens may be static (i.e., not dynamic or electro-active). In some alternative embodiments, either or both may be dynamic using electro-active features.

[0082] In some embodiments, two or more of the waveguides 270, 280, 290, 300, 310 may have the same associated depth plane. For example, multiple waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same depth plane, or multiple subsets of waveguides 270, 280, 290, 300, 310 may be configured to output images set to the same multiple depth planes, one set per depth plane. This may provide advantages for forming tiled images to provide an extended field of view at those depth planes.

[0083] Continue to refer Figure 6 , the out-coupling optical elements 570, 580, 590, 600, 610 can be configured to redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of out-coupling optical elements 570, 580, 590, 600, 610 that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, the light extraction optical elements 570, 580, 590, 600, 610 can be volume or surface features that can be configured to output light at a specific angle. For example, the light extraction optical elements 570, 580, 590, 600, 610 can be volume holograms, surface holograms, and / or diffraction gratings. In some embodiments, features 320, 330, 340, 350 may not be lenses; instead, they may simply be spacers (eg, cladding and / or structures used to form air gaps).

[0084] In some embodiments, the outcoupling optical elements 570, 580, 590, 600, 610 are diffractive features that form a diffraction pattern, or "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOE has a sufficiently low diffraction efficiency that only a portion of the light beam is deflected toward the eye 210 through each intersection of the DOE, while the remainder continues to move through the waveguide via TIR. The light carrying the image information is thus split into a plurality of related exit beams that exit the waveguide at a plurality of locations, and the result is a fairly uniform pattern of exit emissions toward the eye 210 for that particular collimated beam that bounces within the waveguide.

[0085] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may include a polymer dispersed liquid crystal layer in which droplets contain a diffractive pattern in a host medium, and the refractive index of the droplets may be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the droplets may be switched to a refractive index that does not match the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0086] In some embodiments, a camera assembly 630 (e.g., a digital camera, including visible light and infrared light cameras) may be provided to capture images of the eye 210 and / or tissue surrounding the eye 210, for example, to detect user input and / or monitor a physiological state of the user. As used herein, a camera may be any image capture device. In some embodiments, the camera assembly 630 may include an image capture device and a light source to project light (e.g., infrared light) to the eye, which may then be reflected by the eye and detected by the image capture device. In some embodiments, the camera assembly 630 may be attached to the frame 80 ( Fig.9D ) and can be in electrical communication with processing modules 140 and / or 150 that can process image information from camera assembly 630. In some embodiments, one camera assembly 630 can be used for each eye to monitor each eye separately.

[0087] Reference now Figure 7 , shows an example of an outgoing beam output by a waveguide. One waveguide is shown, but it should be understood that the waveguide assembly 260 ( Figure 6 ) may function similarly, where the waveguide assembly 260 includes multiple waveguides. Light 640 is injected into the waveguide 270 at the input surface 460 of the waveguide 270 and propagates within the waveguide 270 by TIR. At the point where the light 640 impinges on the DOE 570, a portion of the light exits the waveguide as an exit beam 650. The exit beam 650 is shown as being substantially parallel, but as discussed herein, depending on the depth plane associated with the waveguide 270, the exit beam 650 may also be redirected at an angle (e.g., to form a diverging exit beam) to propagate to the eye 210. It should be understood that a substantially parallel exit beam may be indicative of a waveguide having an out-coupling optical element that couples light out to form an image that appears to be disposed at a depth plane at a greater distance (e.g., optical infinity) from the eye 210. Other waveguides or other groups of outcoupling optics may output a more divergent exit beam pattern, which would require the eye 210 to accommodate a closer distance to focus it on the retina and would be interpreted by the brain as light coming from a distance closer to the eye 210 than optical infinity.

[0088] In some embodiments, a full color image may be formed at each depth plane by superimposing images of each component color (eg, three or more component colors). Figure 8An example of a stacked waveguide assembly is shown, where each depth plane includes an image formed using multiple different component colors. The illustrated embodiment shows depth planes 240a-240f, but more or fewer depths are also contemplated. Each depth plane may have three or more component color images associated with it, including: a first image G of a first color; a second image R of a second color; and a third image B of a third color. For the diopters (dpt) following the letters G, R, and B, different numbers are used in the figure to represent different depth planes. As an example, the number following each of these letters represents the diopters (1 / m), or the inverse distance of the depth plane from the viewer, and each box in the figure represents a separate component color image. In some embodiments, in order to account for differences in the eye's focus on light of different wavelengths, the precise placement of the depth planes of different component colors may vary. For example, different component color images of a given depth plane may be placed on depth planes corresponding to different distances from the user. Such an arrangement may increase visual acuity and user comfort and / or may reduce chromatic aberration.

[0089] In some embodiments, light of each component color may be output by a single dedicated waveguide, and therefore, each depth plane may have multiple waveguides associated with it. In such embodiments, each box including the letters G, R, or B in the figure may be understood to represent a separate waveguide, and each depth plane may provide three waveguides, wherein each depth plane provides three component color images. Although for ease of description, the waveguides associated with each depth plane are shown adjacent to each other in the figure, it should be understood that in a physical device, the waveguides may all be arranged in a stack with one waveguide per layer. In some other embodiments, multiple component colors may be output by the same waveguide, so that, for example, each depth plane may provide only a single waveguide.

[0090] Continue to refer Figure 8 In some embodiments, G is green, R is red, and B is blue. In some other embodiments, other colors associated with other wavelengths of light (including magenta and cyan) may be used in addition to red, green, or blue, or one or more of red, green, or blue may be replaced.

[0091] It should be understood that references throughout this disclosure to light of a given color will be understood to include light of one or more wavelengths within a range of wavelengths that a viewer perceives as light of the given color. For example, red light may include light of one or more wavelengths within a range of about 620-780 nm, green light may include light of one or more wavelengths within a range of about 492-577 nm, and blue light may include light of one or more wavelengths within a range of about 435-493 nm.

[0092] In some embodiments, the light source 530 ( Figure 6) can be configured to emit light of one or more wavelengths outside the visual perception range of a viewer, for example, infrared and / or ultraviolet wavelengths. In addition, the incoupling, outcoupling, and other light redirection structures of the waveguide of the display 250 can be configured to direct this light out of the display and toward the user's eye 210, for example, for imaging and / or user stimulation applications.

[0093] Reference now Fig. 9A In some embodiments, it may be desirable to redirect light incident on a waveguide to couple the light into the waveguide. Incoupling optical elements may be used to redirect the light and couple the light into its corresponding waveguide. Fig. 9A A cross-sectional side view of an example of a plurality of stacked waveguides or stacked waveguide groups 660 is shown, each stacked waveguide including an incoupling optical element. Each waveguide can be configured to output light of one or more different wavelengths, or light of one or more different wavelength ranges. It should be understood that stack 660 can correspond to stack 260 ( Figure 6 ), and the stacked waveguide 1200 may correspond to a portion of multiple waveguides 270, 280, 290, 300, 310, except that one or more lights from the image injection devices 360, 370, 380, 390, 400 are injected into the waveguide from a location where the light needs to be redirected for coupling.

[0094] The stacked waveguide group 660 shown includes waveguides 670, 680, and 690. Each waveguide includes an associated coupling-in optical element (which may also be referred to as a light input region on the waveguide), for example, a coupling-in optical element 700 disposed on a major surface (e.g., an upper major surface) of the waveguide 670, a coupling-in optical element 710 disposed on a major surface (e.g., an upper major surface) of the waveguide 680, and a coupling-in optical element 720 disposed on a major surface (e.g., an upper major surface) of the waveguide 690. In some embodiments, one or more of the coupling-in optical elements 700, 710, 720 may be disposed on the bottom major surface of the corresponding waveguide 670, 680, 690 (particularly in the case where one or more coupling-in optical elements are reflective, deflecting optical elements). As shown, the coupling-in optical elements 700, 710, 720 may be disposed on the upper major surface (or the top of the next lower waveguide) of their corresponding waveguides 670, 680, 690, particularly in the case where those coupling-in optical elements are transmissive, deflecting optical elements. In some embodiments, the coupling-in optical elements 700, 710, 720 may be disposed in the body of the respective waveguides 670, 680, 690. In some embodiments, as discussed herein, the coupling-in optical elements 700, 710, 720 are wavelength selective such that they selectively redirect light of one or more wavelengths while transmitting light of other wavelengths. Although shown on one side or corner of their respective waveguides 670, 680, 690, it should be understood that in some embodiments, the coupling-in optical elements 700, 710, 720 may be disposed in other areas of their respective waveguides 670, 680, 690.

[0095] As shown, the coupling-in optical elements 700, 710, 720 can be laterally offset from each other. In some embodiments, each coupling-in optical element can be offset so that it receives light without the light passing through another coupling-in optical element. For example, each coupling-in optical element 700, 710, 720 can be configured to be offset from, for example, Figure 6 The different image injection devices 360 , 370 , 380 , 390 , and 400 shown receive light and may be separated (eg, laterally spaced apart) from the other incoupling optical elements 700 , 710 , 720 such that it receives substantially no light from the other incoupling optical elements 700 , 710 , 720 .

[0096] Each waveguide further includes an associated light distribution element, such as a light distribution element 730 disposed on a major surface (e.g., top major surface) of the waveguide 670, a light distribution element 740 disposed on a major surface (e.g., top major surface) of the waveguide 680, and a light distribution element 750 disposed on a major surface (e.g., top major surface) of the waveguide 690. In some other embodiments, the light distribution elements 730, 740, 750 may be disposed on the bottom major surface of the associated waveguides 670, 680, 690, respectively. In some other embodiments, the light distribution elements 730, 740, 750 may be disposed on the top and bottom major surfaces of the associated waveguides 670, 680, 690, respectively; or the light distribution elements 730, 740, 750 may be disposed on different major surfaces of the top and bottom major surfaces of different associated waveguides 670, 680, 690, respectively.

[0097] The waveguides 670, 680, 690 may be spaced apart and separated by, for example, layers of gas, liquid, and / or solid material. For example, as shown, layer 760a may separate waveguides 670 and 680; and layer 760b may separate waveguides 680 and 690. In some embodiments, layers 760a and 760b are formed of a low refractive index material (i.e., a material having a lower refractive index than the material forming the immediately adjacent waveguide in waveguides 670, 680, 690). Preferably, the refractive index of the material forming layers 760a, 760b is 0.05 or more, or 0.10 or less, less than the refractive index of the material forming waveguides 670, 680, 690. Advantageously, the low refractive index layers 760a, 760b may serve as a cladding layer that promotes total internal reflection (TIR) ​​of light passing through the waveguides 670, 680, 690 (e.g., TIR between the top and bottom major surfaces of each waveguide). In some embodiments, the layers 760a, 760b are formed of air. Although not shown, it is understood that the top and bottom of the illustrated waveguide set 660 may include an adjacent cladding layer.

[0098] Preferably, for ease of manufacturing and other considerations, the materials forming waveguides 670, 680, 690 are similar or identical, and the materials forming layers 760a, 760b are similar or identical. In some embodiments, the materials forming waveguides 670, 680, 690 can be different between one or more waveguides, and / or the materials forming layers 760a, 760b can be different, while still maintaining the various refractive index relationships described above.

[0099] Continue to refer Fig. 9A , light rays 770, 780, 790 are incident on the waveguide group 660. It should be understood that light rays 770, 780, 790 may be injected through one or more image injection devices 360, 370, 380, 390, 400 ( Figure 6 ) are injected into waveguides 670, 680, and 690.

[0100] In some embodiments, the light rays 770, 780, 790 have different properties, such as different wavelengths or different wavelength ranges, which may correspond to different colors. The incoupling optical elements 700, 710, 720 each deflect incident light so that the light propagates through a corresponding one of the waveguides 670, 680, 690 by TIR. In some embodiments, the incoupling optical elements 700, 710, 720 each selectively deflect one or more specific wavelengths of light while transmitting other wavelengths to the underlying waveguide and associated incoupling optical element.

[0101] For example, the coupling-in optical element 700 can be configured to deflect light 770 having a first wavelength or wavelength range while transmitting light 780 and 790 having different second and third wavelengths or wavelength ranges, respectively. The transmitted light 780 is incident on and deflected by the coupling-in optical element 710, which is configured to deflect light having the second wavelength or wavelength range. The light 790 is deflected by the coupling-in optical element 720, which is configured to selectively deflect light having a third wavelength or wavelength range.

[0102] Continue to refer Fig. 9A , the deflected light rays 770, 780, 790 are deflected so that they propagate through the corresponding waveguides 670, 680, 690; that is, the coupling-in optical element 700, 710, 720 of each waveguide deflects light into the corresponding waveguide 670, 680, 690 to couple the light into the corresponding waveguide. The light rays 770, 780, 790 are deflected at an angle that causes the light to propagate through the corresponding waveguide 670, 680, 690 via TIR. The light rays 770, 780, 790 propagate through the corresponding waveguide 670, 680, 690 via TIR until they enter the corresponding light distribution element 730, 740, 750 of the waveguide.

[0103] Reference now Fig. 9B , showing Fig. 9A 600, 680, 690. As described above, the in-coupling light rays 770, 780, 790 are deflected by the in-coupling optical elements 700, 710, 720, respectively, and then propagate through TIR in the waveguides 670, 680, 690, respectively. Then, the light rays 770, 780, 790 are incident on the light distribution elements 730, 740, 750, respectively. The light distribution elements 730, 740, 750 deflect the light rays 770, 780, 790 so that they propagate toward the out-coupling optical elements 800, 810, 820, respectively.

[0104] In some embodiments, the light distribution elements 730, 740, 750 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or distribute light to the outcoupling optical elements 800, 810, 820, and in some embodiments also increase the beam or spot size of the light as it propagates to the outcoupling optical elements. In some embodiments, for example, the light distribution elements 730, 740, 750 may be omitted, and the incoupling optical elements 700, 710, 720 may be configured to deflect light directly to the outcoupling optical elements 800, 810, 820. For example, referring to Fig. 9A , the light distribution elements 730, 740, 750 can be replaced by outcoupling optical elements 800, 810, 820, respectively. In some embodiments, the outcoupling optical elements 800, 810, 820 are exit pupils (EP) or exit pupil expanders (EPE), which guide the light in the viewer's eye 210 ( Figure 7 ). It should be understood that the OPE may be configured to increase the size of the eyebox in at least one axis, and the EPE may increase the eyebox in, for example, an axis orthogonal to the axis of the OPE. For example, each OPE may be configured to redirect a portion of the light that strikes the OPE to the EPE of the same waveguide, while allowing the remainder of the light to continue to propagate down the waveguide. After striking the OPE again, another portion of the remaining light is redirected to the EPE, and the remainder of that portion continues to propagate further down the waveguide, and so on. Similarly, after striking the EPE, a portion of the incident light is directed out of the waveguide toward the user, and the remainder of that light continues to propagate through the waveguide until it strikes the EP again, at which time another portion of the incident light is directed out of the waveguide, and so on. Therefore, each time a portion of the light is redirected by an OPE or EPE, a single beam of coupled-in light can be "copied", thereby forming a field of cloned beams, such as Figure 6 In some embodiments, the OPE and / or EPE may be configured to modify the size of the light beam.

[0105] Therefore, reference Fig. 9A and 9BIn some embodiments, the waveguide group 660 includes waveguides 670, 680, 690; coupling-in optics 700, 710, 720; light distribution elements (e.g., OPEs) 730, 740, 750; and outcoupling optics (e.g., EPs) 800, 810, 820 for each component color. The waveguides 670, 680, 690 may be stacked with air gaps / cladding layers present between each. The coupling-in optics 700, 710, 720 redirect or deflect incident light (using different coupling-in optics that receive light of different wavelengths) into their waveguides. The light then propagates at an angle that will result in TIR within the corresponding waveguide 670, 680, 690. In the example shown, light 770 (e.g., blue light) is deflected by first coupling-in optical element 700 and then proceeds to jump to the waveguide, interacting with light distribution element (e.g., OPE) 730 in the manner previously described, and then interacting with coupling-out optical element (e.g., EPs) 800. Light rays 780 and 790 (e.g., green light and red light, respectively) will pass through waveguide 670, where light 780 is incident on coupling-in optical element 710 and is deflected by coupling-in optical element 710. Light 780 then jumps to waveguide 680 via TIR, proceeds to light distribution element (e.g., OPE) 740 reaching it and then to coupling-out optical element (e.g., EP) 810. Finally, light 790 (e.g., red light) passes through waveguide 690 to coupling-in optical element 720 that is incident on waveguide 690. The light incoupling optical element 720 deflects the light 790 so that it propagates through TIR to the light distribution element (e.g., OPE) 750, and then propagates through TIR to the outcoupling optical element (e.g., EP) 820. The outcoupling optical element 820 then finally couples the light 790 out to the viewer, who also receives outcoupled light from the other waveguides 670, 680.

[0106] Fig. 9C Shows Fig. 9A and 9B A top plan view of an example of multiple stacked waveguides. As shown, the waveguides 670, 680, 690 and the light distribution elements 730, 740, 750 associated with each waveguide and the associated outcoupling optical elements 800, 810, 820 may be vertically aligned. However, as discussed herein, the incoupling optical elements 700, 710, 720 are not vertically aligned; instead, the incoupling optical elements are preferably non-overlapping (e.g., laterally spaced apart as seen in the top view). As further discussed herein, this non-overlapping spatial arrangement facilitates one-to-one injection of light from different sources into different waveguides, thereby allowing a particular light source to be uniquely coupled to a particular waveguide. In some embodiments, arrangements including non-overlapping, spatially separated incoupling optical elements may be referred to as shifted pupil systems, and the incoupling optical elements within these arrangements may correspond to sub-pupils.

[0107] Fig.9D An example of a wearable display system 60 is shown, into which the various waveguides and related systems disclosed herein may be integrated. In some embodiments, the display system 60 is Figure 6 The system 250, wherein Figure 6 Some parts of the system 60 are schematically shown in more detail. For example, Figure 6 The waveguide assembly 260 can be part of the display 70.

[0108] Continue to refer Fig.9D , the display system 60 includes a display 70 and various mechanical and electronic modules and systems that support the functions of the display 70. The display 70 can be coupled to a frame 80, which can be worn by a display system user or viewer 90 and is configured to position the display 70 in front of the eyes of the user 90. In some embodiments, the display 70 can be considered as glasses. In some embodiments, a speaker 100 is coupled to the frame 80 and is configured to be located near the ear canal of the user 90 (in some embodiments, another speaker, not shown, can optionally be located near the other ear canal of the user to provide stereo / shapeable sound control). The display system 60 may also include one or more microphones 110 or other devices to detect sound. In some embodiments, the microphone is configured to allow a user to provide input or commands to the system 60 (e.g., selection of voice menu commands, natural language questions, etc.) and / or may allow audio communication with other people (e.g., with other users of similar display systems. The microphone may also be configured as a peripheral sensor to collect audio data (e.g., sounds from the user and / or the environment). In some embodiments, the display system 60 may further include one or more outwardly directed environmental sensors 112, which are configured to detect objects, stimuli, people, animals, locations, or other aspects of the world around the user. For example, the environmental sensors 112 may include one or more cameras, which may be positioned, for example, facing outward to capture images similar to at least a portion of the normal field of view of the user 90. In some embodiments, the display system may also include a peripheral sensor 120a, which may be separate from the frame 80 and attached to the body of the user 90 (e.g., on the head, torso, limbs, etc. of the user 90). In some embodiments, the peripheral sensor 120a may be configured to acquire data representative of the physiological state of the user 90. For example, the sensor 120a may be an electrode.

[0109] Continue to refer Fig.9D, the display 70 is operably coupled to the local data processing and module 140 via a communication link 130 (such as via a wired lead or a wireless connection), and the local data processing and module 140 can be installed in various configurations, such as being fixedly attached to the frame 80, being fixedly attached to a helmet or hat worn by the user, being embedded in a headset, or being removably attached to the user 90 (e.g., in a backpack configuration, in a belt-coupled configuration). Similarly, the sensor 120a can be operably coupled to the local processor and data module 140 via a communication link 120b (e.g., a wired lead or a wireless connection). The local processing and data module 140 may include a hardware processor and a digital memory such as a non-volatile memory (e.g., flash memory or a hard drive), both of which can be used to assist in processing, caching, and storing data. Optionally, the local processor and data module 140 may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include: a) data captured from sensors that may be operably coupled to the frame 80 or attached to the user 90, such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, gyroscopes, and / or other sensors disclosed herein; and / or b) data (including data related to virtual content) acquired and / or processed using the remote processing module 150 and / or remote data repository 160, which data may be transmitted to the display 70 after such processing or retrieval. The local processing and data module 140 may be operably coupled to the remote processing module 150 and the remote data repository 160 via communication links 170, 180 (such as via wired or wireless communication links), so that these remote modules 150, 160 are operably coupled to each other and can be used as resources for the local processing and data module 140. In some embodiments, the local processing and data module 140 may include one or more of an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio, and / or a gyroscope. In some other embodiments, one or more of these sensors may be attached to the frame 80, or may be a stand-alone structure that communicates with the local processing and data module 140 via a wired or wireless communication path.

[0110] Continue to refer Fig.9DIn some embodiments, the remote processing module 150 may include one or more processors configured to analyze and process data and / or image information, including, for example, one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, the remote data repository 160 may include a digital data storage facility that may be available via the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, the remote data repository 160 may include one or more remote servers that provide information to the local processing and data module 140 and / or the remote processing module 150, such as information for generating augmented reality content. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, thereby allowing for completely autonomous use from the remote module. Optionally, an external system (e.g., a system with one or more processors, one or more computers) including a CPU, GPU, etc. may perform at least a portion of the processing (e.g., generating image information, processing data) and providing information to and receiving information from the modules 140, 150, 160, such as via a wireless or wired connection.

[0111] Posture determination

[0112] As described herein, a display device (e.g., Fig.9D The display system 60 described in the foregoing can present virtual content to a user (e.g., a user wearing a display device, such as a user wearing a display 70 coupled to a frame 80). During the presentation of the virtual content, the display device can determine the posture of the display device and the user's head. As described above, the posture can identify the orientation of the display device and / or the user's head, and optionally the position of the display device and / or the user's head. For example, the display device can present virtual content including a virtual document on a real-world table. When the user rotates his / her head around the document or moves closer to or away from the document, the display device can determine his / her head posture. In this way, the display device can adjust the presented virtual content so that the virtual document appears to be a real document on the real-world table. Although the description references virtual content, augmented reality, etc., the display device can be a virtual reality display system and utilize the techniques described herein.

[0113] The display device may utilize an imaging device (e.g., Fig.9DThe imaging device may be, for example, an outward-facing camera fixed to the display device. Thus, the imaging device may obtain images of the real-world environment, and the display device may use these images to determine the posture. The imaging device may obtain images in response to the passage of time (e.g., every 1 / 10, 1 / 15, 1 / 30, 1 / 60 seconds) or in response to detecting that the display device has moved an amount greater than a threshold. For example, the imaging device may obtain a real-time image of the real-world environment (e.g., the sensor of the camera may be configured to always capture image information). In this example, the display device may determine that the input image information has changed by a greater than a threshold amount or is changing at a greater than a threshold rate. As another example, the display device may include sensors such as in an inertial measurement unit (e.g., a magnetometer, a gyroscope, etc.), and the display device may use these sensors to identify whether the display device has moved by a greater than a threshold amount. In this example, the display device may obtain an image based on information detected by the inertial measurement unit. Therefore, the current image may be obtained via the imaging device and may be different from the previous image based on the user's movement. For example, the display device may obtain a continuous image as the user looks around the room.

[0114] As described herein, a display device may track salient points between consecutive images obtained by an imaging device. In some embodiments, the display device may be configured to perform a frame-to-frame tracking process based on an image block. As described above, a salient point may represent a distinguishable visual point, such as a corner point. In order to track a salient point from a previous image to a current image, the display device may project an image block surrounding a salient point in a previous image onto a current image. As described herein, an image block may be an MxN image region surrounding a salient point. For example, a salient point may correspond to a two-dimensional position in a current image, and an image block may be an MxN image region surrounding a two-dimensional position. The display device may then adjust the position of the projected image block to minimize the error or total difference in pixel intensity between the projected image block and the corresponding image region in the current image. Example error minimization processes may include Levenberg-Marquardt, Conjugate Gradient, and the like. A consistent selected position within an image block, for example, the centroid of the projected image block, may be understood as the position of the tracked salient point in the current image. In this way, the display device may identify the movement of a particular visual point of interest (e.g., a salient point, such as a corner point) from a previous frame to a current frame.

[0115] The display device may also be configured to utilize descriptor-based map-to-frame tracking. As described herein, map-to-frame tracking utilizes map information that indicates real-world locations (e.g., three-dimensional locations) of salient points and associated descriptors. For example, the map information may indicate the three-dimensional coordinates of a particular corner point in a real-world environment. If the particular corner point is imaged by the display device and is therefore represented in the current image, the display device may match the representation in the current image with its corresponding real-world location. Fig. 10A Generating map information is described in more detail. In map-to-frame tracking, new salient points may be extracted from image regions in a current image that have tracked salient points that are less than a threshold measurement value (e.g., less than a threshold number or a threshold density). A descriptor may be generated for each salient point, and the descriptor may be matched with the descriptor of the salient point indicated in the map information. As used herein, a descriptor may describe one or more visual elements associated with a salient point. For example, a descriptor may indicate the shape, color, texture, etc. of a salient point. Additionally, a descriptor may describe an area surrounding each salient point (e.g., an MxN pixel area surrounding a salient point). A descriptor may represent, for example, a histogram of an area surrounding a salient point (e.g., an alphanumeric value associated with a histogram), a hash of the area (e.g., a cryptographic hash calculated from the value of each pixel in the MxN pixel area), and the like.

[0116] Therefore, based on the generated descriptors, the display device may match the descriptors generated for each salient point with the descriptors of the salient points indicated in the map information. In this way, the display device may identify the real-world position (e.g., 3D coordinates) corresponding to each salient point in the current image. Therefore, the salient points in the current image may represent the projection of the corresponding 3D real-world coordinates on the 2D image.

[0117] The display device may determine its pose based on these matches. For example, the display device may perform an example pose estimation process, such as perspective n-point (pnp), effective pnp, pnp with random sampling consistency, etc. The display device may then track salient points in subsequent images. For example, the display device may project salient points in the current image onto subsequent images, etc., as described above.

[0118] Fig. 10A A flow chart of an example process 1000 for determining a posture of a display system and a posture of a user's head is shown. In some embodiments, process 1000 may be described as being performed by a display device (e.g., augmented reality display system 60, which may include processing hardware and software and may optionally provide information to an external system of one or more computers for processing, such as to offload processing to an external system and receive information from an external system). In some embodiments, the display device may be a virtual reality display device including one or more processors.

[0119] The display device obtains a current image of a real-world environment (block 1002). The display device may obtain the current image from an imaging device, such as an outward-facing camera fixed to the display device. For example, the outward-facing camera may be positioned in front of the display device to obtain a view similar to the view seen by the user (e.g., a forward-facing view). As described above with respect to Fig.9D As described, the display device may be worn by a user. The display device may optionally utilize two or more imaging devices and acquire images from each device simultaneously (e.g., substantially at the same time). The imaging device may be configured to thereby obtain a stereoscopic image of the real-world environment, which may be used to determine the depth of a location in the image.

[0120] The display device may trigger or otherwise cause the imaging device to obtain a current image based on a threshold amount of time that has passed since a previously obtained image. Thus, the imaging device may obtain images at a specific frequency, such as 10 times per second, 15 times per second, 30 times per second, and the like. Optionally, the specific frequency may be adjusted based on the processing workload of the display device. For example, if the processor of the display device is used at a rate greater than one or more threshold percentages, the specific frequency may be adaptively reduced. Additionally or alternatively, the display device may adjust the frequency based on the motion of the display device. For example, the display device may obtain information indicating a threshold number of previously determined postures and determine changes between postures. Based on the change, the display device may increase the frequency at which the display device obtains images, for example until the measurement of the central tendency is below a specific threshold. In some embodiments, the display device may utilize a sensor such as included in an inertial measurement unit and increase or decrease the frequency according to the estimated motion of the user. In some embodiments, in addition to obtaining the current image based on the threshold amount of time that has passed, the display device may also obtain the current image based on an estimated user movement greater than a threshold amount (e.g., a threshold distance around one or more three-dimensional axes). For example, the display device may utilize an inertial measurement unit to estimate the motion of the user. In some embodiments, the display device may utilize one or more other sensors (e.g., sensors that detect light, color changes, etc.) to determine that information detected by the sensor changes greater than a threshold (e.g., indicating motion) within a threshold amount of time.

[0121] Thus, the current image may be associated with the user's current view. The display device may store the current image, for example, in a volatile or non-volatile memory for processing. In addition, the display device may have a stored image obtained before the current image. As will be described, the current image may be compared with a previous image, and the salient points tracked from the previous image may be compared with the current image. Thus, the display device may store information associated with each salient point in the previous image. For example, the information may include an image block for each salient point, and optionally include the location where the image block appears in the previous image (e.g., the pixel coordinates of the salient point). In some embodiments, instead of storing the complete previous image, the display device may store an image block for each salient point included in the previous image.

[0122] As described above, the image block may represent an image region of size MxN surrounding a salient point (e.g., an imaged salient point). For example, the salient point may be the centroid of the image block. Since the salient point may be a visual point of interest, such as a corner point, in some embodiments, the corner point may be larger than a single pixel. Therefore, the image block may surround the location of the visual point of interest, such as where two lines intersect (e.g., on the keyboard "7", the image block may surround the intersection of a horizontal line and an inclined vertical line). For example, the display device may select a particular pixel as a salient point, and the image block may surround the particular pixel. Additionally, two or more pixels may be selected, and the image block may surround the two or more pixels. As described below, the image block of the previous image may be used to track the associated salient point in the current image.

[0123] The display device projects the tracked salient points from the previous image to the current image (block 1004). As described above, the display device may store information associated with the salient points included in the previous image. Example information may include image blocks surrounding the salient points and information identifying the positions of the image blocks in the previous image. The display device may project each salient point from the previous image onto the current image. As an example, a pose associated with the previous image may be utilized to project each salient point onto the current image. As will be described below, the display device may determine a pose estimate, such as an optical flow. The pose estimate may adjust the pose determined for the previous image, so that an initial projection of the tracked salient points on the current image may be obtained. As will be described, the initial projection may be refined.

[0124] The display device may determine a posture estimate, sometimes referred to as a priori, based on trajectory prediction (e.g., based on previously determined postures) and / or based on an inertial measurement unit, an extended Kalman filter, a visual inertial odometry, etc. With respect to trajectory prediction, the display device may determine a possible direction in which the user is moving. For example, if a previous threshold number of posture determinations indicates that the user is rotating his / her head downward in a particular way, the trajectory prediction may extend the rotation. With respect to the inertial measurement unit, the display device may obtain information indicating an adjustment to the orientation and / or position measured by the sensor of the inertial measurement unit. Thus, posture estimation may enable determination of an initial estimated position corresponding to each tracked salient point in the current image. In addition to posture estimation, the display device may project the salient point using the real-world position of each salient point indicated in the map information. For example, posture estimation may inform the estimated movement of each salient point from a 2D position in a previous image to a 2D position in the current image. This new 2D position may be compared with the map information, and an estimated position of the salient point may be determined.

[0125] The image block of each salient point in the previous image may be compared to an MxN pixel region of the same size in the current image. For example, the display device may adjust the position of the image block projected onto the current image until the photometric error between the image block and the MxN pixel region of the same size in the current image onto which the image block is projected is minimized (e.g., substantially minimized, such as a local or global minimum, or an error below a user-selectable threshold). In some embodiments, the centroid of the MxN pixel region of the current image may be indicated as corresponding to the tracked salient point. Figure 11-12B , the projection-tracked salient points are described in more detail below.

[0126] Optionally, to determine the pose estimate, the display device may minimize the combined photometric cost function of all projected image blocks by changing the pose of the current image. For example, the display device may project an image block associated with each salient point in a previous image onto the current image (e.g., based on an initial pose estimate as described above). The display device may then globally adjust the image blocks, for example by modifying the initial pose estimate, until the photometric cost function is minimized. In this way, a more accurate and precise pose estimate may be obtained. As will be described below, the precise pose estimate may be used as a priori or regularization when determining the pose of the display device. For example, a precise pose estimate may be associated with a cost function so that deviations from the precise pose estimate have an associated cost.

[0127] Thus, the current image may include salient points tracked from the previous image. As will be described below, the display device may identify image regions in the current image that have tracked salient points that are less than a threshold measurement. For example, this may indicate that a user has moved their head to a new location in the real-world environment. In this way, new image regions of the current image that image the new location may not include salient points tracked from the previous image.

[0128] The display device determines whether an image region of the current image includes tracked salient points that are less than a threshold measurement (block 1006). As described above, the display device may determine its pose based on image patch-based frame-to-frame tracking, such as by projecting tracked salient points onto successive images, and optionally in combination with map-to-frame tracking. If one or more image regions of the current image include tracked salient points that are less than a threshold measurement (e.g., a threshold number of salient points or a threshold density of salient points in an image region), map-to-frame tracking may be utilized.

[0129] Fig. 10B An example image region of an example current image is shown. Fig. 10B In the example of , current images 1020A, 1020B, 1020C, and 1020D are shown. These current images may be obtained via a display device, for example, as described above with respect to block 1002. Current image 1020A is shown with an example image region 1022. As described above, the image region encompasses the entirety of the current image. Therefore, the display device may determine whether the current image 1020A as a whole includes tracked salient points that are less than a threshold measurement value.

[0130] In some other embodiments, the current image may be divided into different parts. Fig. 10B The current image 1020B in the example of is divided into a 5x5 grid, with each area in the grid forming a different part. Therefore, the example image area 1024 is one of these parts. The display device can determine whether any of these parts includes a tracked salient point that is less than a threshold measurement value. In this way, since new locations of the real-world environment are included in the current image 1020B, one or more of these parts may include tracked salient points that are less than the threshold measurement value. The size of the grid can be adjusted by the user and / or the display system. For example, the grid can be selected to be 3x3, 7x7, 2x4, etc. Optionally, the grid can be adjusted during operation of the display system, for example, as the user utilizes the display system, the size of various parts of the image will change (e.g., substantially in real time according to processing limitations, accuracy thresholds, pose estimation differences between images, etc.).

[0131] Current image 1020C is shown with example tracked salient points. In this example, image regions may be determined based on the sparsity of the tracked salient points. For example, image regions 1026A and 1026B are shown as surrounding a single tracked salient point. The size of the image region may be user selectable, or a fixed size determined by the system (e.g., an MxN pixel region). The display device may analyze the tracked salient points and determine whether image regions having less than a threshold measurement value may be located in the current image 1020C. For example, image regions 1026A and 1026B have been identified by the display device as including less than a threshold measurement value. Alternatively, the display device may identify image regions including greater than a threshold measurement value, and identify the remaining image regions as image regions including less than a threshold measurement value. For example, image regions 1028 and 1030 have been identified as including tracked salient points greater than a threshold measurement value. Therefore, in this example, the display device may identify any location outside of images 1028 and 1030 as having a tracked salient point less than a threshold measurement value. The display device may then extract new salient points in these external image regions. Optionally, the display device may determine a clustering measure for the locations in the current image. For example, the clustering measure may indicate the average distance of the locations from the tracked salient points. Additionally, the clustering measure may indicate the average number of tracked salient points within a threshold distance of the location. If the clustering measure is below one or more thresholds, the display device may extract new salient points at these locations. Optionally, the display device may extract new salient points in an MxN region around each location.

[0132] The current image 1020D is shown with an example image area 1032. In this example, the image area 1032 may be placed at a specific location of the current image 1020D, such as the center of the current image 1020D. In some embodiments, the example image area 1032 may represent a specific field of view of the user. The image area 1032 may be a specific shape or polygon, such as a circle, an ellipse, a rectangle, etc. In some embodiments, the image area 1032 may be based on an accuracy associated with a lens of the imaging device. For example, the image area 1032 may represent the center of the lens, which is substantially free of distortion introduced at the edge of the lens. Therefore, the display device may identify whether the image area 1032 includes tracked salient points that are less than a threshold measurement value.

[0133] Reference again Fig. 10A, the display device performs map-to-frame tracking (block 1008). The display device may identify whether any image region of the current image includes tracked salient points that are less than a threshold measurement value. As described above, the display device may extract new salient points in the identified image region. For example, the display device may identify the 2D location of the new salient point in the image region. The display device may then receive descriptors for the newly extracted salient points and the existing, tracked salient points. For example, the display device may provide descriptors for the newly extracted and tracked salient points by, for example, generating these descriptors. Alternatively, the display device may provide descriptors by generating descriptors for the newly extracted salient points and by receiving (e.g., obtaining from a memory) descriptors for the tracked salient points. As an example, the display device may utilize previously generated descriptors for tracked salient points. For example, a descriptor may have been generated when each tracked salient point is newly extracted from the image. These descriptors may be matched by the display device with descriptors stored in the map information. Since the map information stores the real-world coordinates of the salient points, the display device may identify the real-world coordinates of each salient point tracked in the current image. The following will refer to Fig.13 Describes performing map-to-frame tracking in more detail.

[0134] As used herein, map information may be generated by a display device. For example, a display device may utilize a stereoscopic imaging device, a depth sensor, a radar, or the like to determine depth information associated with a location in a real-world environment. The display device may periodically update the map information, for example, every threshold number of seconds or minutes. In addition, the map information may be updated based on the identification that the current image (e.g., obtained from a stereoscopic imaging device) is a key frame. As described above, this may be identified based on time, and optionally, may be identified based on the difference between the current image and a previous (e.g., most recent) key frame. For example, if the change in the current image exceeds a threshold, the current image may be identified as a key frame. These key frames may then be analyzed to update the map information.

[0135] With respect to stereoscopic imaging devices, the display device may generate descriptors for salient points in each stereoscopic image. Using known external calibration information, such as the relative pose between the two imaging devices, depth information may be identified. Based on the descriptor matching of salient points between the stereoscopic images and the depth information, real-world coordinates (e.g., relative to a coordinate reference system) may be determined for each salient point. One or more of the descriptors generated for each matched salient point may then be stored. Thus, during map-to-frame tracking, these stored descriptors for real-world salient points may be matched with descriptors of salient points included in a captured image (e.g., a current image). As an example, one of the stereoscopic imaging devices may obtain a current image (e.g., as described in block 1002). The display device may access the map information and match the descriptors generated for the same imaging device in some embodiments with the descriptors of salient points included in the current image. Alternatively, image patch-based photometric error minimization may be used to match salient points between stereoscopic images to determine real-world coordinates to be stored in the map information. The display device may then generate corresponding descriptors for salient (e.g., from one or more of the stereoscopic images) points and use these descriptors to perform map-to-frame tracking. Further description of generating map information is included at least in FIG. 16 and the related description of U.S. Patent Publication 2014 / 0306866, which is incorporated herein by reference in its entirety.

[0136] Continue to refer Fig. 10A , the display device determines a pose based on the descriptor matching (block 1012). As described above, the display device may identify real-world coordinates (e.g., three-dimensional coordinates relative to a particular coordinate reference system) for each salient point included in the current image. The display device may then determine its pose using, for example, a perspective n-point algorithm. Information associated with the imaging device (e.g., intrinsic camera parameters) may be used to determine the pose. Thus, the pose determined by the display device may represent a camera pose. The display device may adjust the camera pose to determine a pose of the user (e.g., block 1016), a pose associated with the front (e.g., the middle) of the display device, and the like. For example, the display device may linearly transform the camera pose based on a known translation or rotation offset of the camera relative to the user.

[0137] In some embodiments, the display device may utilize information obtained from the IMU to determine the posture. For example, the information may be used as a prior or regularization to determine the posture. The display device may therefore use the inertial measurement unit information as a cost function associated with the determination. As an example, the divergence from the inertial measurement unit information may be associated with the cost. In this way, the inertial measurement information may be taken into account and the accuracy of the resulting posture determination may be improved. Similarly, the display device may utilize information associated with an extended Kalman filter and / or a visual inertial odometry.

[0138] Similarly, the display device can utilize information obtained during frame-to-frame tracking as a similar prior or regularization. As described above, the display device can minimize the combined photometric cost function of all image patches by changing the pose of the current image. In this way, an accurate pose estimate can be identified. This accurate pose estimate can be used as a prior in combination with an IMU, an extended Kalman filter, and / or a visual inertial odometry.

[0139] After determining the gesture, the display device may generate an image block for each of the salient points included in the current image. For example, the display device may generate an image block for the salient points newly extracted from the current image, and generate an image block for the salient points tracked from the previous image. Generating the image blocks may include obtaining an MxN pixel area around each salient point in the current image. Optionally, for the salient points tracked from the previous image, the display device may utilize an image block associated with the previous image. That is, when tracking the salient points into the subsequent image, an image block from the previous image (e.g., not the current image) may be utilized in frame-to-frame tracking. Then, the display device may obtain a subsequent image, and blocks 1002-1016 may be repeated for the subsequent image.

[0140] Continue to refer Fig. 10A , the display device determines a pose from the tracked salient points (block 1014). If no image region of the current image includes a tracked salient point that is less than a threshold measurement, the display device may optionally determine its pose based on the tracked salient points. For example, the display device may determine its pose using, for example, a perspective n-point algorithm. Alternatively, the display device may use IMU, extended Kalman filter, visual inertial odometry information as a prior or regularization. Alternatively, if no image region includes a tracked salient point that is less than a threshold measurement, the display device may execute block 1012 without executing block 1014.

[0141] The display device determines a posture of the display device user (block 1016). The posture of the display device may represent a camera posture, such as a posture associated with an imaging device. The display device may adjust the posture based on a known offset of the user relative to the camera. Optionally, the display device may perform initial training while the user wears the display device to, for example, determine an appropriate offset. The training may inform the user of the angle relative to the imaging device and may be used to determine the posture of the display device user. Some examples of methods of performing initial training may be found in U.S. Application No. 15 / 717,747, filed September 27, 2017, the entire contents of which are incorporated herein by reference.

[0142] Fig.11A flow chart of an example process 1100 for frame-to-frame tracking is shown. In some embodiments, process 1100 may be described as being performed by a display device (e.g., display system 60, which may include processing hardware and software, and may optionally provide information to an external system of one or more computers for processing, such as offloading processing to an external system and receiving information from an external system). In some embodiments, the display device may be a virtual reality display device including one or more processors.

[0143] The display device obtains an image block associated with each salient point from a previous image (block 1102). Fig. 10A , the display device may store image blocks for each salient point to be tracked. Therefore, when obtaining a current image, the display device may obtain (e.g., from a stored memory) image block information associated with salient points included in a previous image.

[0144] The display device projects each acquired image block onto the current image (block 1104). Fig. 12A . Fig. 12A Examples of a previous image (eg, image A 1202) and a current image (eg, image B 1204) are shown. Each image is shown to include salient points to be tracked.

[0145] As mentioned above, about Fig. 10A , the display device may determine a pose estimate associated with the current image B 1204. In addition, each salient point included in the previous image A 1202 has a known real-world position or coordinates (e.g., based on map-to-frame tracking previously performed on this image). Therefore, based on these real-world positions, the pose of the previous image A 1202, and the pose estimate, a projection of the salient point on the image B 1204 may be determined. For example, the pose of the previous image A 1202 may be adjusted according to the pose estimate, and based on the real-world position of the salient point in the image A 1202, the salient point may be projected onto the image B 1204 at the 2D position of the image 1204. As shown, the tracked salient point 1208 is associated with the real-world position 1206. Based on the real-world position 1206, the display device has determined that the salient point 1208 is located in the image B 1204 at the initially estimated 2D position. As described above, the pose estimate may optionally be refined via information from an inertial measurement unit, an extended Kalman filter, a visual inertial odometry, or the like.

[0146] Fig. 12BAn image block 1212 associated with a salient point 1208 projected onto image B 1204 is shown. As will be described below, the projected surface image block 1212 may be adjusted in position on image B 1204. The adjustment may be based on reducing an error associated with the projected image block 1212 and a corresponding image region of image B 1204. The error may be, for example, a difference in pixel values ​​(e.g., intensity values) between pixels of the projected image block 1212 and pixels in a corresponding image region of image B 1204, wherein the position of the image block 1208 is adjusted to minimize the difference in values.

[0147] Reference again Fig.11 and 12A , the display device determines an image area in the current image that matches the projected image block (block 1106). Fig. 12B As shown, image block 1212 has been projected onto image B 1204. The projection may represent, for example, an initial estimated position. Then, the display device may adjust the position of the projected image block 1212 on image B 1204 to accurately estimate. For example, the position of the projected image block 1212 may be moved horizontally or vertically by one or more pixels from the initial estimated position. For each adjusted position, the display device may determine the difference between the projected image block 1212 and the same MxN image region of image B 1204 where the image block 1212 is located. For example, the difference in individual pixel values ​​(e.g., intensity values) may be calculated. The display device may adjust image block 1212 based on photometric error optimization. For example, Levenberg-Marquardt, conjugate gradient, etc. may be used to obtain local error minimization, overall error minimization, error below a threshold, etc.

[0148] The display device identifies the tracked salient point in the current image (block 1108). For example, the tracked salient point 1208 may be identified as having a 2D position corresponding to the centroid of the adjusted image block 1212 on image B 1204. Thus, as shown, the salient point 1208 has been tracked from image A 1202 to image B 1204.

[0149] Fig.13 A flow chart of an example process 1300 for map-to-frame tracking is shown. In some embodiments, process 1300 may be described as being performed by a display device (e.g., augmented reality display system 60, which may include processing hardware and software, and may optionally provide information to an external system of one or more computers for processing, such as offloading processing to an external system and receiving information from an external system). In some embodiments, the display device may be a virtual reality display device including one or more processors.

[0150] The display device extracts new salient points from the current image (block 1302). Fig. 10A , the display device can identify an image region of the current image that has tracked salient points that are less than a threshold measurement value. Example image region in Fig. 10B For these identified image regions, the display device may extract new salient points (e.g., identify the locations of the current image showing salient points). For example, for salient points that are corner points, the display device may perform Harris corner detection, accelerated segmentation test feature (FAST) corner detection, etc. on the identified image regions.

[0151] The display device generates a descriptor for each salient point (block 1304). The display device may generate descriptors for (1) tracked salient points (e.g., salient points tracked from a previous image) and (2) newly extracted salient points. As described above, a descriptor may be generated that describes a visual point of interest in the salient point (e.g., as imaged in the current image) or an MxN image region surrounding the salient point. For example, the descriptor may indicate a shape, color, texture, etc. associated with the salient point. As another example, the descriptor may indicate histogram information associated with the salient point.

[0152] The display device projects the real-world location of the salient point onto the current image (block 1306). Fig.14A . Fig.14A The current image B 1204 is shown with newly extracted salient points (e.g., salient point 1402). For example, the display device has determined that the lower portion of the image B 1204 includes tracked salient points that are less than a threshold measure, and has extracted salient points from the lower portion. Fig.14A In the example of , image B1204 includes 7 salient points - 4 tracked salient points and 3 newly extracted salient points

[0153] The display device identifies the real-world positions corresponding to the salient points included in the image B 1204. The identification may be an initial estimate of the real-world positions of the salient points included in the image B 1204. The estimate may be refined based on descriptor matching so that each real-world position of the salient points in the image B 1204 may be accurately determined.

[0154] Regarding the tracked salient point 1208, the display device may recognize that the tracked salient point 1208 may be located within a threshold real-world distance of the real-world location 1206. Since the salient point 1208 is obtained from the previous image A 1202 (e.g., Figures 12A-12B 1206 . Figure 11-12BAs described in FIG. 1 , the salient point has been accurately tracked from the previous image A 1202 to the current image B 1204. Therefore, the display device can recognize that the tracked salient point 1208 corresponds to the real-world position 1206, which corresponds to the salient point that matches it in the previous image A 1202 (e.g., Fig. 12A 1208). Therefore, the display device may compare the descriptors of the real-world salient points within the threshold distance of the position 1206 with the descriptors generated for the tracked salient points 1208. In this way, the real-world salient points may be matched with the tracked salient points 1208.

[0155] With respect to the newly extracted salient point 1402, the display device may identify that the salient point 1402 may be within a threshold real-world distance of the real-world location 1404. For example, the display device may utilize map information and optionally the pose estimate of image B 1204 to identify an initial estimate of the real-world location of the salient point 1402. That is, the display device may access information indicating the pose of the previous image A 1202 and adjust the pose based on the pose estimate. Optionally, the pose estimate may be based on Fig. 10A 1404. The display device may then determine an initial estimate of a real-world position corresponding to the 2D position of the salient point 1402 based on the adjusted pose. As will be described below, the real-world position for the salient point 1402 may be determined through descriptor matching. Thus, the initial estimate (e.g., real-world position 1404) may reduce the number of comparisons between the descriptors for the salient point 1402 and the descriptors of the salient point indicated in the map information.

[0156] Reference again Fig.14A , the display device matches the descriptors of the salient points (block 1308). The display device may compare the descriptors of the salient points indicated in the map information with the descriptors generated for each salient point in the current image (e.g., as described in block 1304) to find an appropriate match.

[0157] As described above, an initial projection of the salient points indicated in the map information onto the current image may be identified. As an example, a plurality of real-world salient points may be close to the real-world location 1404. The display device may compare the descriptors for these plurality of salient points with the descriptors generated for the tracked salient points 1402. Thus, the initial projection may achieve a reduction in the comparisons that need to be performed because it allows the display device to identify possible real-world locations of the salient points 1402. The display device may match the most similar descriptors, for example, based on one or more similarity measures (e.g., differences in histograms, shapes, colors, textures, etc.). In this manner, the display device may determine the real-world location corresponding to each salient point included in the current image B 1204.

[0158] The display device can then determine its posture, such as Fig. 10A Subsequently, the display device may generate an image block for each salient point included in the current image B1204. Since these salient points will be tracked in subsequent images, the image blocks will be used to perform frame-to-frame tracking as described above.

[0159] For example, Fig. 14B Frame-to-frame tracking after map-to-frame tracking is shown. In the example illustration, image B 1204 (e.g., Fig. 12A -B and the current image in 14A-B) now represents the previous image, and image C 1410 represents the current image. In this example, the salient point 1412 is projected onto the current image C 1410. That is, for example, the image block associated with the salient point 1412 generated after the display device determines its posture as described above can be projected onto the current image. Optionally, and as Fig. 10A As described in FIG. 1 , the image block associated with the salient point 1412 may be the same image block as the image block obtained in the previous image A 1202 .

[0160] Thus, the display device may perform frame-to-frame tracking. Similar to the above description, the current image C1410 may then be analyzed, and any image regions of the current image C1410 having tracked salient points that are less than a threshold measurement value may be identified. Map-to-frame tracking may then be performed, and a new pose may be determined.

[0161] Fig.15 A flow chart of an example process for determining head pose is shown. For convenience, process 1500 may be described as being performed by a display device (e.g., display system 60, which may include processing hardware and software, and may optionally provide information to an external system of one or more computers for processing, e.g., offload processing to an external system and receive information from an external system). In some embodiments, the display device may be a virtual reality display device including one or more processors.

[0162] At block 1502, the display device projects the tracked salient points onto the current image. Fig. 10A, a display device may track two-dimensional image positions of salient points between images obtained via one or more imaging devices. For example, a specific corner point may be included (e.g., shown) in a first image at a specific two-dimensional (2D) position of the first image (e.g., one or more pixels of the first image). Similarly, the specific corner point may be included in a second image at a different 2D position of the second image. As described above, the display device may determine that a specific 2D position of the first image corresponds to a different 2D position of the second image. For example, these different 2D positions respectively show specific corner points, and therefore the specific corner point has been tracked between the first image and the second image. The correspondence between the first image and the second image may be determined based on these tracked salient points.

[0163] like Fig.15 As shown, the output of block 1514, which computes the head pose for the previous image, is obtained by the display device for use by block 1502. In addition, the display device obtains the matched salient points and thus uses this information in block 1502. Thus, in this example, the display device may access the previously computed head pose of the user and information associated with the salient points to be tracked from the previous image to the current image. As described above, this information may include the real-world location of the salient point and the corresponding image patch associated with the salient point (e.g., an MxN image region surrounding the salient point in the previous image). As will be described, this information may be used to track the salient points.

[0164] In block 1502, the display device obtains a current image (e.g., as shown above). Fig. 10A ) and projecting the salient points included in the previous image (e.g., shown) onto the current image, or alternatively projecting the salient points from the map onto the current image. The display device may identify an initial estimated position in the current image corresponding to each salient point. For example, Fig. 14B An example salient point 1412 is shown represented at a 2D location of a previous image B 1204. The example salient point 1412 is then projected onto the current image C 1410. For example, the display device may utilize the pose calculated for the previous image (e.g., box 1514), the pose estimate for the current image, and the real-world position of the salient point included in the previous image (e.g., shown). As described above, the pose estimate may be based on trajectory prediction and / or inertial measurement unit (IMU) information. IMU information is optional, but in some embodiments, IMU information may improve pose estimation relative to trajectory prediction or trajectory prediction alone. In some embodiments, the pose estimate may be the same pose as the pose derived from the previous image. The display device may utilize the pose estimate to adjust the pose calculated for the previous image. Since the display device has access to the real-world positions of the salient points, the display device may project these real-world positions onto the two-dimensional current image based on the adjusted pose.

[0165] Therefore, the display device may estimate the 2D position of the current image corresponding to each salient point. As described above, the display device may store an image block for each salient point being tracked. The image block may be an MxN image area around the 2D position of the image, which image area shows the salient point. For example, the image block may extend from the 2D position of the salient point by a set number of pixels in the horizontal direction of the image. Similarly, the image block may extend from the 2D position of the salient point by a set number of pixels in the vertical direction of the image. The display device may obtain an image block associated with each salient point, such as an MxN image area of ​​a previous image around each image block. Each obtained image block may then be projected onto the current image. As an example, an image block associated with a specific salient point may be obtained. The image block may be projected onto the current image, such as the estimated 2D position around the specific salient point. As described above, the 2D position of the projected image block may be adjusted based on minimizing the photometric error. With regard to the example of a specific salient point, the display device may determine the error between the image block and the MxN area of ​​the current image on which the image block is projected. The display device may then adjust the position of the image block (eg, in a vertical and / or horizontal direction) until the error is reduced (eg, minimized), as disclosed herein.

[0166] At block 1504, the display device may optionally refine the pose estimate. As described above, although an initial pose estimate may be determined, the display device may optionally refine the pose estimate. When calculating the head pose, the display device may utilize the refined pose estimate as a prior (e.g., the refined pose estimate may be associated with a cost function).

[0167] like Fig. 10A As described in , the display device can minimize the combined photometric cost function of all projected image blocks by changing the pose estimate of the current image. Due to the changed pose estimate, the estimated 2D position of the current image corresponding to each salient point will be adjusted. Therefore, the image block projected onto the current image can be globally adjusted according to the changed pose estimate. The display device changes the pose estimate until the minimum combined error (e.g., global or local minimum, minimum less than a threshold, etc.) between the projected image block and the corresponding image area of ​​the current image is identified. As shown in process 1500, inertial measurement unit information, extended Kalman information (EKF), visual inertial odometer (VIO) information, etc. can be used as a prediction when refining the pose estimate.

[0168] At block 1506, the display device refines the projected 2D position of the salient point. As described above, the display device may project an image block (e.g., an image region surrounding the salient point in a previous image) onto the current image. The display device may then compare (1) the image block and (2) the MxN image region in the current image onto which the image block is projected. Initially, the display device may compare the image block associated with the salient point with the MxN image region surrounding the salient point in the current image. Subsequently, the display device may adjust the MxN image region in a vertical direction (e.g., upward or downward in the current image) and / or a horizontal direction (e.g., left or right in the current image). For each adjustment, the image block may be compared with the new MxN image region and an error may be determined. For example, the error may represent the sum of pixel intensity differences between corresponding pixels in the image block and the MxN image region (e.g., the difference between the upper left pixel of the image block and the upper left pixel of the image region may be calculated, etc.). According to an error minimization scheme, such as Levenberg-Marquardt as described above, the display device may identify the MxN image region of the current image, which minimizes the error of the image block. The 2D positions in the current image that are enclosed by the identified MxN image area may be identified as salient points associated with the image block. Thus, the display device may track the 2D positions of salient points between the previous image and the current image.

[0169] At block 1508, the display device extracts salient points in the image region having tracked salient points less than a threshold measurement value. Fig. 10A and Fig. 10B The display device may identify an image region in the current image where a new salient point is to be identified. An extraction process such as Harris corner detection may be applied to the current image, and a 2D position in the current image corresponding to the new salient point may be identified.

[0170] At block 1510, the display device then generates descriptors for the salient points included in the current image. The display device may generate descriptors based on 2D positions corresponding to the salient points in the current image. These salient points include salient points tracked from previous images to the current image and salient points newly identified in the current image. As an example, the descriptors for the specific salient points may be generated based on one or more pixels associated with the 2D position of the specific salient point or based on an image area surrounding the 2D position.

[0171] At block 1512, the display device matches the descriptors included in the map information with the generated descriptors. Fig.13As described in, the display device may access map information and match descriptors included in the map information with the generated descriptors. The map information may include real-world coordinates (e.g., 3D coordinates) of the salient points and descriptors associated with these real-world coordinates. Therefore, the match between the map information descriptor and the generated descriptor indicates the real coordinates of the salient point associated with the generated descriptor.

[0172] In order to match the descriptors, the display device may compare the descriptors included in the map information with each descriptor generated for the salient points included in the current image. In order to limit the number of comparisons performed, the display device may estimate the real-world positions of the salient points included in the current image. For example, the salient points tracked from the previous image to the current image have known real-world coordinates. As another example, the real-world coordinates of the salient points newly identified in the current image may be estimated based on the pose estimation of the display device. Therefore, the display device may use these estimated real-world coordinates to identify the portion of the real-world environment estimated to include each salient point. For example, a specific salient point included in the current image may be determined to have the estimated real-world coordinates. The display device may compare the descriptor generated for the specific salient point with the descriptor included in the map information, which is associated with the real-world coordinates within a threshold distance of the estimated real-world coordinates. Therefore, since the display device can centrally compare, the number of comparisons between the descriptors included in the map information and the generated descriptors can be reduced.

[0173] At block 1514, the display device calculates a head pose. As described above, the display device may calculate a head pose based on real-world coordinates of salient points included in the current image and their corresponding 2D positions in the current image. For example, the display device may use camera information (e.g., intrinsic camera parameters) of the imaging device to perform a perspective n-point algorithm. In this manner, the display device may determine a camera pose of the imaging device. The display may then linearly transform the camera pose to determine a head pose of the user. For example, a translation and / or rotation of the user's head relative to the camera pose may be calculated. The display device may then use the user's head pose for subsequent images, for example, the head pose may be utilized in block 1502.

[0174] Optionally, the display device may utilize an accurate pose estimate as a priori when calculating head pose, as described in block 1504. Additionally, the display device may utilize inertial measurement unit information, extended Kalman filter information, inertial visual odometry information, etc. as a priori.

[0175] Computer vision for detecting objects in the surrounding environment

[0176] As described above, the display system can be configured to detect objects in the environment around the user or attributes of the environment. Detection can be accomplished using a variety of techniques, including various environmental sensors (e.g., cameras, audio sensors, temperature sensors, etc.), as discussed herein. For example, an object can represent a salient point (e.g., a corner point).

[0177] In some embodiments, computer vision techniques may be used to detect objects present in the environment. For example, as disclosed herein, a forward-facing camera of a display system may be configured to image the surrounding environment, and the display system may be configured to perform image analysis on the image to determine the presence of objects in the surrounding environment. The display system may analyze images acquired by an outward-facing imaging system to perform scene reconstruction, event detection, video tracking, object recognition, object pose estimation, learning, indexing, motion estimation, or image restoration, among other things. As other examples, the display system may be configured to perform facial and / or eye recognition to determine the presence and position of a face and / or human eye in the user's field of view. One or more computer vision algorithms may be used to perform these tasks. Non-limiting examples of computer vision algorithms include: scale-invariant feature transform (SIFT), speeded-up robust features (SURF), orientation FAST and rotate BRIEF (ORB), binary robust invariant scalable keypoints (BRISK), fast retinal keypoints (FREAK), Viola-Jones algorithm, Eigenfaces method, Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm, visual simultaneous localization and mapping (vSLAM) technology, sequential Bayesian estimator (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive threshold (and other threshold techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histogram, various machine learning algorithms (such as support vector machine, k-nearest neighbor algorithm, naive Bayes, neural network (including convolutional or deep neural network), or other supervised / unsupervised models, etc.), etc.

[0178] One or more of these computer vision techniques may also be used with data acquired from other environmental sensors (such as, for example, microphones) to detect and determine various properties of objects detected by the sensors.

[0179] As discussed herein, objects in the surrounding environment may be detected based on one or more criteria. When the display system detects the presence of the criteria in the surrounding environment using a computer vision algorithm or using data received from one or more sensor components (which may or may not be part of the display system), the display system may then signal the presence of the object.

[0180] Machine Learning

[0181] Various machine learning algorithms can be used to learn to recognize the presence of objects in the surrounding environment. Once trained, the machine learning algorithm can be stored by the display system. Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (e.g., ordinary least squares regression), instance-based algorithms (e.g., learning vector quantization), decision tree algorithms (e.g., classification and regression trees), Bayesian algorithms (e.g., naive Bayes), clustering algorithms (e.g., k-means clustering), association rule learning algorithms (e.g., a-priori algorithms), artificial neural network algorithms (e.g., perceptrons), deep learning algorithms (e.g., deep Boltzmann machines or deep neural networks), dimensionality reduction algorithms (e.g., principal component analysis), integration algorithms (e.g., cascade generalization) and / or other machine learning algorithms. In some embodiments, each model can be customized for each data group. For example, a wearable device can generate or store a basic model. The basic model can be used as a starting point to generate additional models specific to data types (e.g., specific users), data groups (e.g., groups of additional images obtained), conditional situations, or other variants. In some embodiments, the display system can be configured to utilize a variety of techniques to generate models for analyzing aggregated data. Other techniques may include using predefined thresholds or data values.

[0182] The criteria for detecting an object may include one or more threshold conditions. If the analysis of the data acquired by the environmental sensor indicates that the threshold condition has been passed, the display system may provide a signal indicating the presence of an object in the surrounding environment. The threshold condition may involve quantitative and / or qualitative measurements. For example, the threshold condition may include a score or percentage associated with the possibility of the object being present in the environment. The display system may compare the score calculated from the data of the environmental sensor with the threshold score. If the score is above a threshold level, the display system may detect the presence of a reflection and / or an object. In some other embodiments, if the score is below a threshold, the display system may signal the presence of an object in the environment. In some embodiments, the threshold condition may be determined based on the user's emotional state and / or the user's interaction with the surrounding environment.

[0183] It should be understood that each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings may be embodied in and fully or partially automated by: code modules executed by one or more physical computing systems, hardware computer processors, and dedicated circuits; and / or electronic hardware configured to execute specific and specific computer instructions. For example, a computing system can include a general-purpose computer (e.g., a server), a dedicated circuit, etc., programmed with specific computer instructions or a dedicated computer. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpretive programming language. In some embodiments, specific operations and methods may be performed by circuits specific to a given function.

[0184] Furthermore, certain embodiments of the functionality of the present disclosure may be mathematically, computationally, or technically complex enough that dedicated hardware or one or more physical computing devices (with appropriate proprietary executable instructions) may be necessary in order to perform the functionality (e.g., due to the amount or complexity of the computations involved) or to provide results in substantially real time. For example, a video may include multiple frames, each with millions of pixels, and in order to process the video data to provide a desired image processing task or application in a commercially reasonable amount of time, dedicated programmed computer hardware may be necessary.

[0185] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer memory, including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile memory, and combinations of the same and / or similar elements. In some embodiments, the non-transitory computer-readable medium may be part of one or more of the local processing and data module (140), the remote processing module (150), and the remote data repository (160). The methods and modules (or data) may also be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based media and wire / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). The results of the disclosed processes or processing steps may be stored persistently or otherwise in any type of non-transitory physical computer memory, or may be transmitted via a computer-readable transmission medium.

[0186] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood as potentially representing a code module, code segment or code portion, which includes one or more executable instructions that implement a specific function (such as a logical function or an arithmetic function) or step in the process. Various processes, frames, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed according to the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any specific order, and the frames, steps or states associated therewith can be executed in other appropriate orders, such as in serial, parallel or some other manner. Tasks or events can be added to or removed from the disclosed example embodiments. In addition, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products.

[0187] The foregoing description has described the present invention with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0188] In fact, it will be understood that the systems and methods of the present disclosure each have several innovative aspects, any single aspect of which is not solely responsible for or necessary for the desired properties disclosed herein. The various features and processes described above can be used independently of each other or can be used in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure.

[0189] Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as being performed in certain combinations, or even initially claimed as such, in some cases, one or more features from a claimed combination can be deleted from that combination, and a claimed combination may involve sub-combinations or variations of sub-combinations. No single feature or group of features is necessary or indispensable for every embodiment.

[0190] It will be understood that conditional terms used herein, such as (among other things) "can", "can", "may", "may", "for example", etc., are generally intended to express that certain embodiments include and other embodiments do not include certain features, elements and / or steps, unless otherwise specifically stated or otherwise understood in the context. Therefore, such conditional terms are generally not intended to imply that features, elements and / or steps are necessary for one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining whether these features, elements and / or steps are included in or will be performed in any specific embodiment with or without programmer input or prompting. The terms "include", "comprise", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive meaning (rather than its exclusive meaning), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, the articles "a", "an", and "said" used in the present application and the appended claims should be interpreted as meaning "one or more" or "at least one", unless otherwise specifically stated. Similarly, although the operations may be depicted in a particular order in the accompanying drawings, it should be appreciated that such operations do not need to be performed in the particular order or in a sequential order, or all illustrated operations are performed to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown can be incorporated into the schematically illustrated example methods and processes. For example, one or more additional operations can be performed before, after, simultaneously or during any of the illustrated operations. In addition, in other embodiments, the operations may be rearranged or reordered. In some cases, multitasking and parallel processing may have advantages. In addition, the separation of various system components described in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims can be performed in different orders and still achieve the desired result.

[0191] Thus, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the widest scope consistent with this disclosure, the principles and novel features disclosed herein.

Claims

1. An augmented reality display system, comprising: one or more imaging devices; one or more processors; as well as One or more computer storage media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: obtaining, via the one or more imaging devices, a current image of a real-world environment, the current image comprising a plurality of points for determining a gesture; projecting a first image patch-based salient point from a previous image onto a corresponding point among the plurality of points in the current image; extracting a second salient point from the current image; providing corresponding descriptors for the salient points, wherein the salient points include the first salient point and the second salient point; matching the salient points associated with the current image to real-world locations specified in a descriptor-based map of the real-world environment; and determining a gesture associated with the system based on the matching, the gesture indicating at least an orientation of the one or more imaging devices in the real-world environment, The step of extracting the second salient point comprises: determining that an image region of the current image has less than a threshold number of salient points projected from the previous image, wherein the image region comprises a subset of the current image; and One or more descriptor-based salient points are extracted from the image region, and the extracted salient points include the second salient point.

2. The augmented reality display system according to claim 1, wherein: The operation further includes adjusting a position of the first salient point based on the image block on the current image, wherein adjusting the position includes: obtaining a first image block associated with the first salient point, wherein the first image block includes a portion of the previous image containing the first salient point and a region of the previous image surrounding the first salient point; and A second image block similar to the first image block is located in the current image, wherein the first salient point is located at a position in the second image block similar to the first image block.

3. The augmented reality display system according to claim 2, wherein: Positioning the second image block includes minimizing a difference between the first image block in the previous image and the second image block in the current image.

4. The augmented reality display system according to claim 2, wherein: Projecting the patch-based first salient point onto the current image is based at least in part on information from an inertial measurement unit of the system.

5. The augmented reality display system according to claim 1, wherein: The system is configured to adjust a size associated with the subset based on one or more processing constraints or a difference between one or more previously determined postures.

6. The augmented reality display system according to claim 1, wherein: Matching salient points associated with the current image to real-world locations specified in the map of the real-world environment comprises: accessing map information including real-world locations of salient points and associated descriptors; and Descriptors for salient points in the current image are matched with descriptors for salient points at real-world locations.

7. The augmented reality display system according to claim 5, wherein: The operations further include: The salient points provided in the map information are projected onto the current image, wherein the projection is based on one or more of an inertial measurement unit, an extended Kalman filter, or a visual inertial odometry.

8. The augmented reality display system according to claim 1, wherein: The system is configured to generate the map using at least the one or more imaging devices.

9. The augmented reality display system according to claim 1, wherein: Determining the pose is based on the real-world positions of salient points and relative positions of the salient points in a view captured in the current image.

10. The augmented reality display system according to claim 1, wherein: The operations further include: An image block is generated that is associated with a corresponding salient point extracted from the current image, such that, for a subsequent image of the current image, the image block includes the salient point available for being projected onto the subsequent image.

11. The augmented reality display system according to claim 1, wherein: Providing the descriptor includes generating a descriptor for each of the salient points.

12. An augmented reality display system, comprising: one or more imaging devices; One or more processors, the processors configured to: obtaining a current image of the real-world environment; performing frame-to-frame tracking on the current image so that image block-based salient points included in a previous image are projected onto the current image; extracting one or more new salient points from an image region of the current image when the image region has less than a threshold number of image patch-based salient points projected from the previous image, wherein the image region includes a subset of the current image; performing map-to-frame tracking on the current image so that descriptor-based salient points included in a map database are matched with salient points in the current image; and A gesture associated with the display system is determined.

13. A method for augmenting a reality display system, comprising: obtaining, via one or more imaging devices, a current image of a real-world environment, the current image including a plurality of points for determining a gesture; projecting a first image patch-based salient point from a previous image onto a corresponding point among the plurality of points in the current image; extracting a second salient point from the current image; providing corresponding descriptors for the salient points, wherein the salient points include the first salient point and the second salient point; matching the salient points associated with the current image to real-world locations specified in a descriptor-based map of the real-world environment; and determining a gesture associated with a display system based on the matching, the gesture indicating at least an orientation of the one or more imaging devices in a real-world environment, The step of extracting the second salient point comprises: determining that an image region of the current image has less than a threshold number of salient points projected from the previous image, wherein the image region comprises a subset of the current image; and One or more descriptor-based salient points are extracted from the image region, and the extracted salient points include the second salient point.

14. The method according to claim 13, further comprising adjusting the position of the first salient point based on the image block on the current image, wherein: Adjusting the position comprises: obtaining a first image block associated with the first salient point, wherein the first image block includes a portion of the previous image containing the first salient point and a region of the previous image surrounding the first salient point; and A second image block similar to the first image block is located in the current image, wherein the first salient point is located at a position in the second image block similar to the first image block.

15. The method according to claim 14, wherein: Positioning the second image block includes determining an image block in the current image that has the smallest difference from the first image block.

16. The method according to claim 14, wherein: Projecting the patch-based first salient point onto the current image is based at least in part on information from an inertial measurement unit of the display system.

17. The method according to claim 13, further comprising: A size associated with the subset is adjusted based on one or more processing constraints or differences between one or more previously determined postures.

18. The method according to claim 13, wherein: Matching salient points associated with the current image to real-world locations specified in the map of the real-world environment comprises: accessing map information including real-world locations of salient points and associated descriptors; and Descriptors for salient points in the current image are matched with descriptors for salient points at real-world locations.

19. The method according to claim 18, further comprising: The salient points provided in the map information are projected onto the current image, wherein the projection is based on one or more of an inertial measurement unit, an extended Kalman filter, or a visual inertial odometry.

20. The method according to claim 13, wherein: Determining the pose is based on the real-world positions of salient points and relative positions of the salient points in a view captured in the current image.

21. The method of claim 13, further comprising: An image block is generated that is associated with a corresponding salient point extracted from the current image, such that, for a subsequent image of the current image, the image block includes the salient point available for being projected onto the subsequent image.

22. The method according to claim 13, wherein: Providing the descriptor includes generating a descriptor for each of the salient points.

23. The method of claim 13, further comprising generating the map using at least the one or more imaging devices.

Images