Tracking the optical flow of backscattered laser speckle patterns

Abstract

The present invention relates to tracking the optical flow of backscattered laser speckle patterns. An augmented reality helmet includes a transparent display that allows a user to simultaneously view the real world and virtual content located in the real world, and also includes at least one source of coherent light and at least one sensor array for sensing the speckle pattern produced when the coherent light illuminates an ambient surface over a series of times. A circuit is provided for sensing a shift in the speckle pattern and determining the motion that caused the shift in the speckle pattern and adjusting the display of a virtual object displayed by the augmented reality helmet to compensate for the motion.

Description

[0001] This application is a divisional application of the application with application number 201880026770.X, application date April 23, 2018, and invention name “Tracking optical flow of backscattered laser spot patterns”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is based on provisional patent application serial number 62 / 489,339 filed on April 24, 2017. Technical Field

[0004] The present invention relates to an augmented reality helmet. Background Art

[0005] Recently, virtual reality devices have been proposed that immerse a user in a computer-generated virtual world. Certain virtual reality devices include a pair of displays placed in close proximity to the user's eyes and corrective optics interposed between the pair of displays and the user's eyes, the purpose of which is to allow the user to focus on images displayed on the displays despite the close proximity of the displays. The main application of such virtual reality devices is gaming, although other applications, such as scientific data visualization, may also be considered.

[0006] A technology related to virtual reality but more complex that is being developed is augmented reality. An augmented reality wearable device (i.e., a headgear with a glasses form factor) will allow a user to simultaneously view the real world and virtual, computer-generated content superimposed on the real world. In order to improve the illusion that the virtual content is real and / or more seamlessly integrate the virtual content and the real world, it is desirable that the virtual content appear to exist in an inertial reference frame that is fixed to the user's real environment, even though the user rotates their head with the helmet, or walks around in the surrounding environment. Thus, for example, if the virtual content includes a virtual book placed on the corner of a real-world table, the book should remain fixed in the real world, for example, on the corner of the table, even if the user rotates their head with the augmented reality helmet, which can project an image of the book on the corner of the table. To do this, the movement of the helmet carried on the user's head needs to be carefully tracked, and the image generated by the augmented reality helmet will quickly shift in the field of view to eliminate the effect of the helmet's movement. One way to track the direction of an object is to use one or more gyroscopes. However, gyroscopes inherently drift, so attempting to fix a virtual object in the real world based on gyroscope output can cause the virtual object to slowly drift relative to the real world when the intention is to maintain a fixed position relative to the real world. Summary of the invention

[0007] According to certain embodiments disclosed herein, an augmented reality helmet is equipped with at least one laser and at least one optical sensor array. Each laser emits a light beam that is partially reflected from a room surface (e.g., a wall, ceiling, or floor), thereby producing a speckle pattern detected by the associated optical sensor array. The optical sensor array can be of the type that can be used in a digital camera, for example, but in the present application, the 2D optical sensor array does not need to be used to capture a focused image of an object, but rather they can capture a speckle pattern generated by a laser that is reflected from the room surface. The movement of the speckle pattern on the sensor array (appropriately referred to as "optical flow" in some cases) is used to calculate the movement of the augmented reality helmet in an inertial reference frame fixed to the room. The image displayed by the eyepiece using augmented reality is offset according to the calculated movement to maintain its position or velocity in the inertial reference frame fixed to the room.

[0008] One aspect of the present disclosure is an augmented reality helmet, comprising: at least one source of image-modulated light; at least one transparent eyepiece, which is configured to couple the image-modulated light into the user's eyes while allowing the user to see the real world; at least a first coherent light source directed outward from the augmented reality helmet in a first direction so as to project coherent light onto at least one environmental surface; at least a first sensor array, which is configured to receive light diffracted by and reflected from at least one environmental surface, the light forming a first speckle pattern on the first sensor array; an electronic circuit, which is coupled to the source of image-modulated light and the first sensor array and is configured to: operate the source of image-modulated light to inertial The method comprises: displaying a virtual object at a set of coordinates defined in a reference system, the inertial reference system being fixed to a physical space occupied by an augmented reality helmet worn by a user; receiving a first copy of a first speckle pattern at a first time; receiving a second copy of the first speckle pattern at a second time; determining an offset of the second copy of the first speckle pattern relative to the first copy of the first speckle pattern; determining the movement of the augmented reality helmet in the physical space occupied by the user based on the offset of the second copy of the first speckle pattern relative to the first copy of the first speckle pattern; and then adjusting the image modulated light based on the movement of the augmented reality helmet to compensate for the movement of the augmented reality helmet and keep the virtual object at a set of coordinates defined in the inertial reference system. In addition, a second sensor array may be provided, the second sensor array being configured to receive light reflected from at least one environmental surface. In addition, the augmented reality may also include a second coherent light source, the second coherent light source being directed outward from the augmented reality helmet in a second direction so as to project coherent light onto at least one environmental surface. In addition, the augmented reality helmet may also include at least one pupil stop, the at least one pupil stop being configured to substantially exclude light from the second coherent light source reflected by at least one environmental surface from reaching the first sensor array. In addition, the augmented reality helmet may further include at least one optical component configured to establish mutually exclusive emission solid angle ranges of the first coherent light source and the second coherent light source. In addition, the first sensor array may be installed to have a first field of view, and the second sensor array may be installed to have a second field of view, and the first field of view may partially overlap with the second field of view.

[0009] One aspect of the present disclosure is a method of sensing and distinguishing translational motion of a structure along a set of three independent axes and rotation of the structure about one of the set of three independent axes, the method comprising: providing at least one source of coherent light that emits light within a predetermined solid angle range and is coupled to the structure; providing a first 2D optical sensor array coupled to the structure, the first 2D optical sensor array having a first normal vector pointing in a first direction; providing a second 2D optical sensor array coupled to the structure, the second 2D optical sensor array having a second normal vector pointing in a second direction, wherein the first normal vector and the second normal vector define a plane and the first normal vector is angled in the plane relative to the second normal vector; illuminating a non-specular ambient surface with at least one source of coherent illumination, thereby generating a first speckle pattern at the first 2D optical sensor array and a second speckle pattern at the 2D optical sensor array; sensing translation of the structure along a first of the set of three independent axes by sensing a first optical flow of the first speckle pattern on the first 2D optical sensor array and sensing a second optical flow of the second speckle pattern on the second 2D optical sensor array, The method comprises sensing a translation of the structure along a second of the three independent axes by sensing a third optical flow of the first speckle pattern on the first 2D optical sensor array and sensing a fourth optical flow of the second speckle pattern on the second 2D optical sensor array, comprising a non-zero projection on the plane outside the angular range between the first normal vector and the second normal vector, wherein the third optical flow and the fourth optical flow have a common directional projection on the difference vector between the first normal vector and the second normal vector; sensing a translation of the structure along a third of the three independent axes by sensing the same sensed perpendicular directional optical flow of the first speckle pattern on the first 2D optical sensor array and the second speckle pattern on the second 2D optical sensor array, comprising a non-zero component perpendicular to the plane; and sensing a rotation of the structure about a first of the three independent axes by sensing opposite perpendicular directional optical flows of the first speckle pattern on the first 2D optical sensor array and the second speckle pattern on the second 2D optical sensor array.

[0010] One aspect of the present disclosure includes a method of sensing and distinguishing translational motion of a structure along a set of three independent axes and rotation about the set of three independent axes, the method comprising: providing a first 2D optical sensor array coupled to the structure, the first 2D optical sensor array having a first surface normal pointing in a first direction and a first field of view; providing a second 2D optical sensor array coupled to the structure, the second 2D optical sensor array having a second surface normal pointing in a second direction and a second field of view; providing a third 2D optical sensor array coupled to the structure, the third 2D optical sensor array having a third surface normal pointing in a third direction and a third field of view; wherein the first 2D optical sensor array is coupled to the structure, the first 2D optical sensor array having a first surface normal pointing in a first direction and a first field of view; The first direction, the second direction and the third direction are independent; providing at least one coherent light source, the coherent light source projects light into the first field of view, the second field of view and the third field of view, wherein the light diffracted by and reflected from the non-specular surrounding surface forms a first speckle pattern array on the first 2D optical sensor, a second speckle pattern on the second 2D optical sensor array and a third speckle pattern on the third 2D optical sensor array; by using the second 2D optical sensor array to sense the translation of the second speckle pattern in a direction at an azimuth angle relative to the first direction, and using the third 2D optical sensor array to sense the direction of the third speckle pattern at an azimuth angle relative to the first direction The method comprises the steps of: sensing a rotation about a first direction by sensing a translation of the second speckle pattern in the first direction by using the second 2D optical sensor array to sense a translation of the second speckle pattern in the first direction, and using the third 2D optical sensor array to sense a translation of the third speckle pattern in the third direction, thereby sensing a translation in the first direction; sensing a rotation about the second direction by using the first 2D optical sensor array to sense a translation of the first speckle pattern in a direction at an azimuth angle relative to the second direction, and using the third 2D optical sensor array to sense a translation of the third speckle pattern in a direction at an azimuth angle relative to the second direction, thereby sensing a rotation about the second direction; sensing a translation of the first speckle pattern in the second direction by using the first 2D optical sensor array to sense a translation of the first speckle pattern in ... thereby sensing a translation of the first speckle pattern in the third direction, thereby sensing a rotation about the second direction; sensing a translation of the first speckle pattern in the second direction by using the first 2D optical sensor array to sense a translation of the first speckle pattern in the second direction, thereby sensing a translation of the third speckle pattern in the third direction, thereby sensing a rotation about the second direction; sensing a translation of the first speckle pattern in the second direction by using the first 2D optical sensor array to sense a translation of the first speckle pattern in the second direction, thereby sensing a translation of the third speckle pattern in the third direction, thereby sensing a rotation about the second direction; sensing a translation of the first speckle pattern in the second direction by using the first 2D optical sensor array to sense a translation of the first speckle pattern in the second direction, thereby sensing a translation of the third speckle The method comprises the steps of: sensing a rotation around a third direction by using the second 2D optical sensor array to sense the translation of the second speckle pattern in a direction forming an azimuth relative to the third direction, and sensing the translation in the second direction by using the third 2D optical sensor array to sense the translation of the third speckle pattern in the second direction; sensing a rotation around the third direction by using the second 2D optical sensor array to sense the translation of the second speckle pattern in a direction forming an azimuth relative to the third direction, and using the first 2D optical sensor array to sense the translation of the first speckle pattern in a direction forming an azimuth relative to the third direction; and sensing the translation in the third direction by using the first 2D optical sensor array to sense the translation of the first speckle pattern in the third direction, and using the second 2D optical sensor array to sense the translation of the second speckle pattern in the third direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings illustrate the design and utility of preferred embodiments of the present invention, wherein similar elements are represented by common reference numerals. In order to better understand how the above and other advantages and objects of the present invention are obtained, the invention briefly described above will be described in more detail by reference to specific embodiments of the invention shown in the drawings. It should be understood that these drawings depict only typical embodiments of the invention and therefore should not be considered as limiting the scope of the invention. The invention will be described and explained with additional specificity and detail through the use of the drawings, in which:

[0012] Figure 1 is a front view of a user wearing an augmented reality helmet according to an embodiment disclosed herein;

[0013] Figure 2 is included in Figure 1 A schematic diagram of a motion detection system in an augmented reality helmet is shown;

[0014] Figure 3 yes Figure 2 A schematic diagram of a portion of a motion detection system is shown, illustrating optical isolation between two coherent light sources according to one embodiment;

[0015] Figures 4a-4f A series of speckle patterns corresponding to a series of translation steps is shown;

[0016] Figure 5 is a schematic diagram of a motion detection system included in an augmented reality helmet according to an alternative embodiment;

[0017] Figure 6 is a perspective view of an augmented reality helmet having a glasses form factor according to another embodiment;

[0018] Figure 7 is included in Figure 6 Schematic diagram of the motion detection system in the augmented reality helmet shown in 6;

[0019] Figure 8 According to one embodiment, it is applicable to Figure 1 and Figure 6 A block diagram of an augmented reality helmet of the augmented reality helmet shown;

[0020] Fig. 9 The operation according to one embodiment Figure 1 A flowchart of the method for augmenting reality helmet shown; and

[0021] Fig.10 is a schematic diagram of an example of using an augmented reality helmet according to one embodiment. DETAILED DESCRIPTION

[0022] Figure 11 is a front view of a user's head 102 wearing an augmented reality helmet 104 according to one embodiment. The augmented reality helmet 104 has a glasses form factor and includes a frame 106 in which are mounted a left (user's left) eyepiece 108 and a right eyepiece 110 in the form of lenses (eyepieces) of the helmet 104. The eyepieces 108, 110 may include surface relief reflective and transmissive diffraction gratings that control the redirection of light toward the user's eyes, or may include tilted partially reflective mirrors embedded in the bulk eyepieces 108, 110. For example, each eyepiece 108, 110 may include an internal coupling grating (ICG) that receives the image-modulated light and diffracts the image-modulated light into an angle above the critical angle for total internal reflection (TIR) ​​within each eyepiece. In addition to the ICG, each eyepiece may also include an orthogonal pupil expansion grating (OPE) that gradually deflects light directed by the ICG to the OPE downward toward an exit pupil expander (EPE), which gradually deflects the light out toward the user's eye position. Alternatively, the eyepieces 108, 110 may include a partially reflective front surface coating that redirects light toward the user's eyes. In the latter case, the eyepiece need not be planar, but may include curved front and rear (near the eye) surfaces with finite optical power. Again referring to Figure 1 , a left source 112 of image-modulated light is optically coupled to the left eyepiece 108, and a right source 114 of image-modulated light is optically coupled to the right eyepiece 110. The sources 112, 114 of image-modulated light may, for example, include a liquid crystal on silicon (LCoS) image projector, a fiber scanner, or a light-emitting (e.g., micro light-emitting diode, micro organic light-emitting diode) display panel. The eyepieces 108, 110 are used to optically couple the image-modulated light to the user's eyes. The eyepieces 108, 110 may also be transparent, thereby allowing the user to view the real world. The sources 112, 114 of image-modulated light and the eyepieces 108, 110 are used to display images, which are referred to as "virtual content" in the present context. The virtual content augments the real world visible through the eyepieces 108, 110. The left eyepiece 108 is combined with the left source 112 of image-modulated light to form a left display, and the right eyepiece 110 is combined with the right source 114 of image-modulated light to form a right display.

[0023] For many applications, it is desirable to position or move virtual content (e.g., a virtual person walking) in an inertial reference frame that is fixed to the environment (e.g., a room) in which the user of the augmented reality helmet is located, despite the fact that the augmented reality helmet 104 moves and rotates with the user as the user walks in the environment and turns his or her head 102 to look in different directions. To achieve the latter purpose, the images displayed through the eyepieces 108, 110 and the sources 112, 114 of image-forming modulated light must be offset to compensate for the user's movements. Determining the correct offset requires careful tracking of the user's movements (translation and rotation). To this end, the augmented reality helmet 104 is equipped with an upward-pointing laser (coherent illumination source) 116, a sideways-pointing laser 118, and a forward-pointing laser 120, all of which are mechanically coupled to the frame 106. The pointing directions of the lasers 116, 118, 120 can also be changed relative to the aforementioned directions, and these are just simple examples of appropriate directions. An upward facing 2D optical sensor array 122, a sideward facing 2D optical sensor array 124, and a forward facing 2D optical sensor array 126 are also provided and mechanically coupled to the frame 106. The 2D optical sensor arrays 122, 124, 126 may include, for example, complementary metal oxide semiconductor (CMOS) pixel arrays or charge coupled device (CCD) pixel arrays. The light emitted by the upwardly directed laser 116 is angled within the field of view of the upwardly directed 2D optical sensor array 122, the light emitted by the sideward directed laser 118 is angled within the field of view of the sideward facing 2D optical sensor array 124, and the light emitted by the forward directed laser 120 is angled within the field of view of the forward facing 2D optical sensor array 126. Although the light from each laser 116, 118, 120 is angled within the field of view of a particular one of the 2D optical sensor arrays 122, 124, 126, to the extent that the light is propagating away from the transmitting laser and the associated 2D optical sensor array 122, 124, 126, it will not be detected by the associated 2D optical sensor array 122, 124, 126 to any significant extent unless it is scattered back by the surface it strikes (e.g., a wall, ceiling, or furniture). With the exception of windows and mirrors, such surfaces are typically diffusely reflective, so the scattered back light will be in the form of a speckle pattern that fills the space between the scattering surface and the 2D optical sensor array 122, 124, 126. In this case, the speckle pattern is the diffraction pattern of the small-scale roughness of the surface. The speckle pattern is detected by the 2D optical sensor array 122, 124, 126. Furthermore, when the user moves his head 102 with the augmented reality helmet 104, the 2D optical sensor arrays 122, 124, 126 will move through the space filled with the speckle pattern, and such movement is detected by reading out the 2D optical sensor arrays 122, 124, 126 at successive times.Because the lasers 116, 118, 120 move with the 2D optical sensor arrays 122, 124, 126, the motion of the speckle pattern detected on the 2D optical sensor arrays will be twice the physical motion of the 2D optical sensor arrays 122, 124, 126. In the case of rigid body mechanics, there are three translational degrees of freedom (e.g., translation along the Cartesian X, Y, Z axes) and three rotational degrees of freedom (e.g., yaw, pitch, and roll). For convenience, we can also use variables Rx, Ry, Rz to refer to the rotations around the X, Y, and Z axes.

[0024] Figure 2 is included in Figure 1 Schematic diagram of a motion detection system 200 in an augmented reality helmet is shown. The motion detection system 200 includes an upwardly pointing laser 116, a sideways pointing laser 118, a forwardly pointing laser 120, an upwardly facing 2D optical sensor array 122, a sideways facing 2D optical sensor array 124, and a forwardly facing 2D optical sensor 126. The aforementioned components are shown as being oriented relative to a 3D Cartesian coordinate system 202, which includes a Z axis (upward), a Y axis (sideways, to the user's left), and an X axis (forward). Coherent light from the upwardly facing (+Z) laser 116 is incident on a non-specular ceiling surface 204, which includes some small-scale surface roughness that diffracts the coherent light to form a spot pattern that is incident on the upwardly facing (+Z) 2D optical sensor array 122. Similarly, coherent light from the sideward (+Y) facing laser 118 is incident on a patch of the sideward facing wall 206, which scatters the light, thereby forming a speckle pattern on the sideward (+Y) facing 2D optical sensor array 124. Likewise, coherent light from the forward (+X) facing laser 120 is incident on a patch of the rearward facing wall 208, which scatters the light, thereby forming a speckle pattern on the forward (+X) facing 2D optical sensor array 126. Figure 2, the symbols Rx, Ry, Rz represent rotations about the X, Y, and Z axes. The component of the translational motion of the augmented reality helmet 104 along a particular Cartesian axis (X, Y, or Z) results in a spot shift that is parallel to the component on the two 2D optical sensor arrays 122, 124, and / or 126 that face in directions different from the component of the translational motion. Thus, for example, a translational component in the +X direction results in a +X direction translation of the spot pattern on the 2D optical sensor array 122 facing upward (+Z) and the 2D optical sensor array 124 facing sideways (+Y). The sensed translation of the spot pattern is equal to twice the physical translation of the helmet 104. Each 2D optical sensor array 122, 124, 126 operates at a sufficiently high frame rate so that the laser spot sensed by the 2D sensor array 122, 124, 126 moves a distance between consecutive frames that is a fraction of the size of the spot (i.e., full width half maximum (FWHM)).

[0025] like Figure 2 As shown in the schematic diagram of , the 2D optical sensor arrays 122, 124, 126 are displaced from the Cartesian coordinate system origin. In the augmented reality helmet 104, the 2D optical sensor array is displaced from the effective center of rotation of the head movement, which is located near the back of the user's neck. By sensing the shift of the speckle pattern produced on the 2D optical sensor arrays 122, 124, 126, rotations about the X, Y, and Z axes can be sensed. The component of the rotation about a given axis (X, Y, or Z) will cause the speckle shift to be azimuth relative to the given axis on the 2D optical sensor arrays 122, 124, or 126 facing in a direction other than parallel to the given axis. (For present purposes, positive rotational sense about a positive Cartesian axis is defined using the right-hand rule.) Thus, for example, the -X direction is angled azimuthally relative to the Z axis in the lateral direction (+Y) facing the 2D optical sensor array 124, and the +Y direction is also angled azimuthally relative to the Z axis in the forward direction (+X) facing the 2D optical sensor array 126. Thus, a positive Rz rotational component about the Z axis will cause an X-direction spot shift in the lateral direction (+Y) facing the 2D optical sensor array 124, and will cause a -Y-direction spot shift in the forward direction (+X) facing the 2D optical sensor array 126.

[0026] The following matrix equation EQU.1 relates incremental translations along the Cartesian X, Y, and Z axes and incremental rotations about the Cartesian X, Y, and Z axes to the offset of the speckle pattern on the 2D optical sensor arrays 122 , 124 , 126 .

[0027] EQU.1

[0028]

[0029] The non-zero coefficients in the coefficient matrix on the left side of EQU.1 have three subscripts. The first subscript identifies one of the six degrees of freedom of the augmented reality helmet 104, namely, X, Y, Z translation and rotation around the X, Y and Z axes Rx, Ry, Rz. The second subscript identifies the direction (X, Y or Z) that one of the 2D optical sensor arrays 122, 124, 126 faces, which is equal to the direction in which the normal vector points to the front surface (light reception) of the 2D optical sensor array 122, 124, 126. The third subscript identifies the direction (X, Y or Z) of the spot offset on the specific 2D optical sensor array 122, 124, 126 identified by the second subscript. The values ​​of the non-zero coefficients in the first three columns of the coefficient matrix associated with the translational degree of freedom are 2. The values ​​of the non-zero coefficients in the third to sixth columns of the translation matrix are 2 / R sensor , where R sensor is the distance between the 2D optical sensor array 122, 124, or 126 identified by the second subscript and the effective center of rotation (e.g., the back of the user's neck) when the user wears the augmented reality helmet 104.

[0030] The column vector on the left side of EQU.1 includes incremental translations Δx, Δy, Δz as the first three elements, and incremental rotations ΔRx, ΔRy, ΔRz about the X, Y, and Z axes as the last three elements. Each element of the column vector on the right side of EQU.1 is a spot offset on one of the 2D optical sensor arrays 122, 124, 126. Each spot offset element is represented by the letter S with a subscript and a superscript. The subscript identifies the direction in which one of the 2D optical sensor arrays 122, 124, 126 has its normal vector oriented (the direction it faces). The superscript identifies the direction of the spot offset on the 2D optical sensor arrays 122, 124, 126. By way of illustration, the first row of the coefficient matrix C indicates the translation in the y direction (e.g., C ). y，x，y ) and rotation around the Z axis (as shown in C Rz，x，y As shown) both will result in a y-direction (relative to the z-axis azimuth) spot shift on the 2D optical sensor array 126 facing the forward (+X) direction. The coefficient matrix is ​​easily inverted, resulting in the following matrix equation EQU.2:

[0031] EQU.2

[0032]

[0033] Among them, C -1is the inverse of the matrix in EQU.1. EQU.2 is used to determine the incremental translation and rotation of the augmented reality helmet 104 based on the vector of spot shifts appearing on the left side of EQU.2. The incremental spot shift can be obtained by reading the spot pattern formed on the 2D optical sensor array 122, 124, 126 at two consecutive times and determining the relative shift of the spot pattern at two consecutive times. Optical flow tracking methods such as the Farneback method or normalized cross correlation can be used to determine the relative shift of the spot pattern. Based on the determined incremental translation and rotation, the virtual content output via the source 112, 114 of imaging modulated light and the eyepiece 108, 110 is adjusted to maintain position and / or motion in an inertial coordinate system fixed to the environment in which the augmented reality helmet 104 is used. The 2D optical sensor arrays 122, 124, 126 are read at a sufficiently high rate, compared to the maximum expected rate of the augmented reality helmet, so that the frame-by-frame change in the pointing angle of the lasers 116, 118, 120 is a fraction of the FWHM beam divergence of the lasers 116, 118, 120 (including Figure 3 The diffusers 310, 312 of the 2D optical sensor arrays 122, 124, 126 are used as diffusers. Therefore, the frame-by-frame changes in the speckle pattern sensed by the 2D optical sensor arrays 122, 124, 126 are mainly offsets (optical flow). In addition, the lasers 116, 118, 120 can be operated in a pulsed mode with a pulse rate equal to the frame rate of the 2D optical sensor arrays 122, 124, 126 and a pulse width substantially shorter than the frame period (1 / frame rate) of the 2D optical sensor arrays 122, 124, 126. Using such a short laser pulse width will help avoid blurring caused by motion of the speckle pattern.

[0034] Figure 3 yes Figure 2 A schematic diagram of a portion of a motion detection system is shown, illustrating optical isolation between two coherent light sources according to one embodiment. Figure 3The illustrated view includes a side-pointing laser 118 and a forward-pointing laser 120. As shown, the side-pointing laser 118 includes a first laser diode 302 optically coupled to a first collimating lens 304, and similarly, the forward-pointing laser 120 includes a second laser diode 306 optically coupled to a second collimating lens 308. The collimating lenses 304, 308 establish mutually exclusive solid angle ranges of coverage (emission) of the side-pointing laser 118 and the forward-pointing laser 120. The first collimating lens 304 forms a beam that propagates laterally, and the second collimating lens 308 forms a beam that propagates forward. Alternatively, instead of a collimating lens, a lens that forms a controlled divergent beam can be used. In addition, a beam shaping lens that establishes a certain radial (e.g., flat top) or non-axisymmetric beam profile can be used instead of a collimating lens. A first low-angle diffuser 310 is located in front of the first collimating lens 304, and a second low-angle diffuser 312 is located in front of the second collimating lens 308. The low angle diffusers 310, 312 reduce the brightness of the beam formed by the collimating lenses 304, 308 and are useful for eye safety. Note that the laser diodes 302, 306 may be infrared emitting, in which case the user would not be able to see the emitted light. As a non-limiting example, the low angle diffusers 310, 312 may be characterized by a diffuse FWHM of 2° to 20°. Figure 3Also shown in FIG. 1 are a side-facing 2D optical sensor array 124 and a front-facing 2D optical sensor array 126. A side-facing pupil stop 314 is located in front of the side-facing 2D optical sensor array 124 and is used to limit the field of view of the side-facing 2D optical sensor array 124. Similarly, a front-facing pupil stop 316 is located in front of the front-facing 2D optical sensor array 126 and is used to limit the field of view of the front-facing 2D optical sensor array 126. The side-facing pupil stop 314 establishes a field of view of the side-facing 2D optical sensor array 124 that substantially covers the solid angle range of the side-pointing laser 118 emission expanded by the first low-angle diffuser 310, and that substantially excludes the solid angle range of the forward-pointing laser 120 emission expanded by the second low-angle diffuser 312. Similarly, the forward-facing pupil stop 316 establishes a field of view for the forward-facing 2D optical sensor array 126 that substantially covers the solid angle range emitted by the forward-pointing laser 120 expanded by the second low-angle diffuser 312, and substantially excludes the solid angle range emitted by the side-pointing laser 118 expanded by the first low-angle diffuser 310. Therefore, each 2D optical sensor array will only receive a single speckle pattern produced by the light emitted by its associated laser. Another purpose of the pupil stops 314, 316 is to enlarge the size of the spots in the speckle pattern incident on each 2D optical sensor array 124, 126. The characteristic size of the spots in the speckle pattern should be equal to or larger than the size of the individual sensor elements (pixels) that make up the 2D sensor arrays 124, 126. Although in Figure 3 Not shown, the upwardly directed laser 116 may have Figure 3 The same internal design of the side-pointing laser 118 and the forward-pointing laser 120 is shown, and is also equipped with a low-angle diffuser, and the upward-facing 2D optical sensor array 122 may also be equipped with a pupil stop.

[0035] According to another embodiment, the sensor arrays 122, 124, 126 are spectrally isolated from the emissions of the lasers 116, 118, 120, rather than one being associated with them. In one implementation, the upwardly directed laser 116 emits a first spectral line having a first peak wavelength, the sideways directed laser 118 emits a second spectral line having a second peak wavelength, and the forward directed laser 120 emits a third spectral line having a third peak wavelength. A first spectral selective filter that transmits the first spectral line but not the second or third spectral line is located above the 2D optical sensor array 122 facing upward; a second spectral selective filter that transmits the second spectral line but not the first or third spectral line is located above the 2D optical sensor array 124 facing sideways; and a third spectral selective filter that transmits the third spectral line but not the first or second spectral line is located above the 2D optical sensor array 126 facing forward.

[0036] Figure 4a is an initial image of the speckle pattern 400, which can be used to track Figure 1-3 The motion in the system shown. Figure 4b-4f Shows Figure 4a A series of images 408, 410, 412, 414, 416 of the speckle pattern 400 are shown after a series of translation increments. An initial arbitrary position 402 in the speckle pattern 400 is marked by a vertical line 404 and a horizontal line 406. Optical flow tracking methods such as, but not limited to, those described above may be used to track the motion of the speckle pattern. The translation distance of the speckle pattern is equal to twice the physical translation of the augmented reality helmet 104 including the motion detection system 200. At each successive Figure 4b-4f The position of an arbitrarily selected position 402 is shown in FIG.

[0037] Figure 5 is a schematic diagram of a motion detection system 500 included in an augmented reality helmet 104 according to an alternative embodiment. The system 300 includes a first laser 506 oriented to point in a first direction, a second laser 508 oriented to point in a second direction, and a third laser oriented to point in a third direction. The system 300 also includes a first 2D optical sensor array facing the first direction, a second 2D optical sensor array facing the second direction, and a third 2D optical sensor array facing the third direction. Figure 3 In the case of the motion detection system 300 shown, the lasers 116, 118, 120 are pointed in a set of three orthogonal directions, and the 2D optical sensor arrays 122, 124, 126 are oriented to face the same set of three orthogonal directions. Figure 5 In the case of the motion detection system 500 shown, the first, second and third directions are not orthogonal.

[0038] Figure 6 is a perspective view of an augmented reality helmet 600 having a glasses form factor according to another embodiment, Figure 7 6 is a schematic diagram of a motion detection system 700 included in an augmented reality helmet 600. The augmented reality helmet 600 includes a frame 602 that supports a left (user's left) transparent eyepiece 604 and a right transparent eyepiece 606. The left transparent eyepiece 604 includes a left ICG 604A, a left OPE 604B, and a left EPE 604C, and similarly, the right transparent eyepiece 606 includes a right ICG 606A, a right OPE 606B, and a right EPE 606C. Each of the eyepieces 604, 606 may alternatively include multiple waveguides to process multiple color channels and or output images with different wavefront curvatures (corresponding to different virtual image distances). A left source 608 of imaging modulated light is optically coupled to the left transparent eyepiece 604, and a right source 610 of imaging modulated light is optically coupled to the right transparent eyepiece 606. The transparent eyepieces 604, 606 are used to optically couple the imaging modulated light to the user's eyes.

[0039] The augmented reality helmet 600 is also equipped with a right laser 611, a left laser 612, a left 2D optical sensor array 614, and a right 2D optical sensor array 616. The forward direction corresponds to Figure 6 The +X axis of the Cartesian coordinate system triad is shown. The left laser 612 and the left 2D optical sensor array 614 face a direction tilted to the left (rotated about the vertical Z axis) relative to the forward direction, and the right laser 611 and the right 2D optical sensor array 616 face a direction tilted to the right relative to the forward direction. The direction facing the left laser 612 and the left optical sensor 614 is determined by the left normal vector N perpendicular to the light receiving surface of the left optical sensor 614. L is defined by, and the direction in which the right laser 611 and the right optical sensor 616 face is defined by the right normal vector N perpendicular to the light receiving surface of the optical sensor 616. RL Alternatively, there may be a difference between the directions each laser and associated sensor faces. The left normal vector N L and the right normal vector N R The difference vector D between Figure 7 6. The left 2D optical sensor array 614 and the right 2D optical sensor array 616 have separate fields of view. The field of view of the left 2D optical sensor array 614 includes at least a substantial portion of the emission range of the left laser 612, while the field of view of the right 2D optical sensor array 616 includes at least a substantial portion of the solid angle range of the emission of the right laser 611. The fields of view of the left and right 2D optical sensor arrays 614, 616 may be limited by pupil stops or other field of view limiting optical components ( Figure 6For example, the field of view of the left 2D optical sensor array 614 can be limited to exclude the angular range of the right laser 611 emission, and vice versa.

[0040] Optionally, the left 2D optical sensor array 614 may be equipped with a left imaging lens 618, and the right 2D optical sensor array 616 may be equipped with a right imaging lens 620. The imaging lenses 618, 620 focus and magnify or reduce the spot light from the left focal plane 622 and the right focal plane 624 in the space in front of the imaging lenses 618, 620 to be incident on the 2D optical sensor arrays 614, 616, respectively.

[0041] It should be noted that the Y axis of the Cartesian coordinate triad extends laterally from left to right. The motion detection system 700 incorporated in the augmented reality helmet 600 is capable of sensing and distinguishing four degrees of freedom, including translational components along the X-axis, Y-axis, and Z-axis, and rotation about the forward-facing X-axis. The left normal vector N L and the right normal vector N R A virtual plane is defined, and N L and N R On the limited plane, N L and N R There is an angular range between them. L and N R The speckle patterns on the left and right 2D optical sensor arrays 614, 616 of the defined plane are offset in opposite directions to sense the L and N R N on the limited plane L and N R In the aforementioned case, the first optical flow on the left 2D optical sensor array 614 and the second optical flow on the right 2D optical sensor array 616 have projections in opposite directions on the difference vector D.

[0042] On the other hand, by sensing the L and N R The speckle patterns on the left and right 2D optical sensor arrays 614, 616 of the defined plane are offset in the same direction to sense the L and N R On the limited plane in N L and N R In the latter case, the first optical flow on the left 2D optical sensor array 614 and the second optical flow on the right 2D optical sensor array 616 have a common direction projection on the difference vector D. In addition, by sensing the direction perpendicular to the direction indicated by N L and N RThe speckle patterns of the same direction on the left and right 2D optical sensor arrays 614, 616 of the defined plane are offset to sense the vertical direction of N L and N R In addition, the translation component of the plane defined by N can be detected by sensing the L and N R The speckle patterns in opposite perpendicular directions on the defined planar left and right 2D optical sensor arrays 614, 616 are offset to sense rotation about the forward extending +X axis.

[0043] Figure 8 is a block diagram of an augmented reality helmet 900 according to one embodiment. Figure 8 The design shown in FIG. 1 may be used in accordance with certain embodiments of the present invention. Figure 1 and Figure 6 The augmented reality helmet 900 includes a left light-transmitting (see-through) eyepiece 902, a right light-transmitting eyepiece 904, a first laser diode 906, a first 2D optical sensor array 908, an Nth laser diode 910, and an Nth 2D optical sensor array 912, all of which are mechanically coupled to an augmented reality helmet frame 914. It should be understood that the use of the identifier "Nth" indicates that the number of similar components including the component identified as "Nth" is variable. For example, in Figure 1 In the case of the augmented reality helmet 104 shown in FIG, three laser diodes are utilized in the three lasers 116, 118, 120, and Figure 6 In the case of the augmented reality helmet 600 shown in FIG, a single laser diode ( Figure 6 not shown).

[0044] A left source 916 of image modulated light is optically coupled to the left light transmitting eyepiece 902, and a right source 918 of image modulated light is optically coupled to the right light transmitting eyepiece 904. The sources 916, 918 of image modulated light may, for example, include a fiber scanner, an LCoS projector, or a MEMS beam scanner, or a micro-emitting display. A left image data source 920 is coupled to the left source 916 of image modulated light, and a right image data source 922 is coupled to the right source 918 of image modulated light. The image data sources 920, 922 may, for example, be in the form of a display driver. The left source 916 of image modulated light is combined with the left light transmitting eyepiece 902 to form a left display 948, and the right source 918 of image modulated light is combined with the right light transmitting eyepiece 904 to form a right display 950. The left and right sources 916, 918 of image modulated light modulate light according to data provided by the left and right image data sources 920, 922, respectively. The left and right image data sources 920, 922 may take the form of frame buffers provided with data by a graphics processing unit (GPU), in conjunction with a microprocessor running a game engine program.

[0045] The first laser diode 906 is optically coupled to the first diffuser 926 through the first collimating lens 924, and the Nth laser diode 912 is optically coupled to the Nth diffuser 930 through the Nth collimating lens 928. Coherent light from the first laser diode 906 coupled through the first collimating lens 924 and the first diffuser 926 is incident on a first surface patch 932 (e.g., a wall, ceiling, floor, furniture) in the environment of the augmented reality helmet 900, which forms a first speckle pattern (a diffraction pattern of small-scale roughness of the surface) incident on the first 2D optical sensor array 908. Similarly, coherent light from the Nth laser diode 910 coupled through the Nth collimating lens 928 and the Nth diffuser 930 is incident on an Nth surface patch 934 in the environment of the augmented reality helmet 900, which forms an Nth speckle pattern incident on the Nth 2D optical sensor array 912.

[0046] The first sensor readout circuit 936 is coupled to the first 2D optical sensor array 908, and the Nth sensor readout circuit 938 is coupled to the Nth 2D optical sensor array 912. An inertial measurement unit (IMU) 952 is mechanically coupled to the frame 914. The first sensor readout circuit 936, the Nth sensor readout circuit 938, the left image data source 920, the first image data source 922, the IMU 952, at least one processor 940, at least one program memory 942, and at least one workspace memory 944 are coupled together via at least one bus 946. The at least one processor 940 may include, for example, a microprocessor, a graphics processing unit, a digital signal processor, and / or a microcontroller. The IMU 952 may be used in conjunction with the above-described components for detecting motion via speckle optical flow. For example, the IMU 952 may be used as an additional redundant source of motion information to improve accuracy, or information from the IMU may be combined with information obtained via speckle flow monitoring to fully determine the 6DoF of the helmet 900.

[0047] Fig. 9 The operation according to one embodiment Figure 1 A flowchart of method 1000 for an augmented reality helmet 104 is shown. Figure 6 , 8 The augmented reality helmets 600, 900 shown in FIG. 1 and 2 may operate in a similar manner. In block 1002, a surface (e.g., 204, 206, 208) in the local environment of the helmet 104 is illuminated with lasers 116, 118, 120 fixed to the augmented reality helmet 104. In block 1004, the display of the augmented reality helmet is operated to display at least one virtual object at a location (defined by a set of coordinates) in the local environment. The eyepieces 108, 110 form a display together with the sources 112, 114 of image-forming modulated light. Fig.10 yes Figure 1 The basis shown Fig.10 A schematic diagram of an example of the use of an augmented reality helmet 104 is shown using the method. Fig.10 , a helmet 1100 is displayed on the user's head 102. The user is viewing a virtual object 1102 in the form of a book, which is displayed using the display of the augmented reality helmet. The virtual object 1102 is located on a real table 1104. Other examples of virtual content may include, for example, people, animals, fictional creatures, and / or moving objects. In the case of moving virtual content, the movement of the virtual content is defined in an inertial reference frame fixed to the physical environment, and the movement of the head device is tracked so that it can be compensated (cancelled) so that the movement of the head device does add velocity to the expected movement of the virtual content relative to the physical environment. Referring again to Fig. 9 In block 1006, light sensor arrays 122, 124, 126 are used to sense a speckle pattern generated by lasers 116, 118, 120 scattered by environmental surfaces (e.g., 204, 206, 208). In performing blocks 1006 and 1010 described below, for example, at least one processor 940 ( Figure 8 ) under the control of workspace memory 944 ( Figure 8 ) from the sensor readout circuits 936, 938 ( Figure 8) receives the speckle pattern. Block 1008 marks the start of a loop that is executed for each successive time. In block 1010, the light sensor array is again used to sense the speckle pattern generated by the lasers 116, 118, 120 scattered by the environmental surfaces (e.g., 204, 206, 208). In block 1012, the offset of each specific speckle pattern compared to at least one previous measurement of the specific speckle pattern is determined. The optical flow determination method described above can be used in executing block 1012. In block 1014, the change in translation and rotation coordinates of the augmented reality head device is determined based on the offset of the speckle pattern determined in block 1012. In executing block 1014, EQU.2 above can be used. Next, in block 1016, as determined in block 1014, the position (coordinates) of the virtual object on the display of the augmented reality helmet 104 is offset according to the change in at least one coordinate so as to maintain the position of the virtual object in the local environment, for example, to maintain the position of the virtual book 1102 on the real table 1104. The position of the virtual object is offset by adjusting the image-forming modulated light coupled to the user's eye through the eyepiece. The method 1100 may be performed under the control of a program stored in at least one memory 942 and executed by at least one processor 940 using at least one workspace memory 944. More generally, the head device 100, 600, 900 is provided with some form of electronic circuitry, which may optionally include an application specific integrated circuit (ASIC) and / or an FPGA as a non-limiting example, which determines the spot offset between consecutive times and determines the incremental motion (translation and rotation) between consecutive times based on the spot offset, and adjusts the position of the displayed virtual object to compensate for the incremental motion.

[0048] Although the above embodiments include augmented reality glasses that include a transparent eyepiece through which a user can view the real world while also viewing virtual content, alternatively, the above 6DoF tracking system can be incorporated into a virtual reality goggle in which the user's field of view of the real world is blocked and the user can only see virtual content. The above 6DoF system can also be applied to an augmented reality in which the user cannot directly view the real world, but can view images of the real world captured by one or more cameras and displayed to the user together with virtual content.

[0049] Various example embodiments of the present invention are described herein. Reference is made to these examples in a non-limiting sense. They are provided to illustrate the more widely applicable aspects of the present invention. Various changes may be made to the described invention and equivalents may be substituted without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt specific circumstances, materials, compositions of matter, processes, process actions or steps to the purpose, spirit or scope of the present invention. In addition, as will be appreciated by those skilled in the art, each individual variant described and illustrated herein has discrete components and features that may be easily separated or combined with the features of any one of the other several embodiments without departing from the scope or spirit of the present invention. All of these modifications are intended to fall within the scope of the claims associated with this disclosure.

[0050] The present invention includes methods that can be performed using subject devices. The method can include an act of providing such suitable devices. Such providing can be performed by an end user. In other words, the "providing" act requires only the acquisition, access, processing, locating, setting, activating, powering on, or other action by the end user to provide the necessary devices in the method. The methods described herein can be performed in any order of the events described that is logically possible and in the order of events described.

[0051] The exemplary aspects of the present invention and the details about material selection and manufacturing have been described above. For other details of the present invention, it can be understood in conjunction with the patents and publications referenced above and what is generally known or understood by those skilled in the art. This is also true about the method-based aspects of the present invention in terms of additional actions as usually or logically adopted.

[0052] In addition, although the present invention has been described with reference to several examples that optionally include various features, the present invention is not limited to being described or represented as expected for each deformation of the present invention. Without departing from the actual spirit and scope of the present invention, various changes can be made to the described present invention, and can be replaced by equivalents (whether stated herein or not included for the purpose of simplicity). In addition, if the range of values ​​is provided, it should be understood that each intermediate value between the upper and lower limits of the range and or any other intermediate value in the range of this description is included in the present invention.

[0053] In addition, it is contemplated that any optional features of the described inventive variations may be stated and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there may be a plurality of the same item. More specifically, as used herein and in the related claims, the singular forms "one," "said," and "the" include plural objects unless specifically stated. In other words, in the above description and in the claims related to the present disclosure, the use of articles allows for "at least one" target item. It is also important to note that such claims may be drafted to exclude any optional elements. Therefore, this statement is intended to be used as a priori basis for the use of such exclusive terms as "alone," "only," etc., in conjunction with the expression of the claim elements, or for the use of "negative" limitations.

[0054] Without using such exclusive terms, the term "comprising" in the claims related to the present disclosure should allow the inclusion of any other elements, regardless of whether a given number of elements are listed in such claims, or the addition of features can be regarded as changing the nature of the elements described in the claims. Except as specifically defined herein, all technical terms used herein should be given the most commonly understood meaning while maintaining the validity of the claims.

[0055] The present invention is not limited to the examples provided and / or this description, but is limited only by the scope of the claim language associated with this disclosure.

Claims

1. A method of sensing and distinguishing translational motion of a structure along a set of three independent axes and rotation of the structure about one of the set of three independent axes, the method comprising: providing at least one coherent light source that emits light within a predetermined solid angle range and is coupled to the structure; providing a first 2D optical sensor array coupled to the structure, the first 2D optical sensor array having a first normal vector pointing in a first direction; providing a second 2D optical sensor array coupled to the structure, the second 2D optical sensor array having a second normal vector pointing in a second direction, wherein the first normal vector and the second normal vector define a plane and wherein the first normal vector is angled relative to the second normal vector in the plane; illuminating a non-specular ambient surface using at least one coherent light source, thereby producing a first speckle pattern at the first 2D optical sensor array and producing a second speckle pattern at the 2D optical sensor array; sensing a translation of the structure along a first of the set of three independent axes by sensing a first optical flow of the first speckle pattern on the first 2D optical sensor array and sensing a second optical flow of the second speckle pattern on the second 2D optical sensor array, the first optical flow and the second optical flow having opposite direction projections on a difference vector between the first normal vector and the second normal vector; sensing translation of the structure along a second of the set of three independent axes by sensing a third optical flow of the first speckle pattern on the first 2D optical sensor array and sensing a fourth optical flow of the second speckle pattern on the second 2D optical sensor array, which includes a non-zero projection on the plane outside an angular range between the first normal vector and the second normal vector, wherein the third optical flow and the fourth optical flow have a common directional projection on the difference vector between the first normal vector and the second normal vector; sensing translation of the structure along a third of the set of three independent axes including a non-zero component perpendicular to the plane by sensing the same sensed perpendicular direction optical flow of the first speckle pattern on the first 2D optical sensor array and the second speckle pattern on the second 2D optical sensor array; and The rotation of the structure about the first of the three independent axes is sensed by sensing the opposite perpendicular optical flows of the first speckle pattern on the first 2D optical sensor array and the second speckle pattern on the second 2D optical sensor array. 2 . The method of claim 1 , wherein the at least one coherent light source comprises a first laser disposed on a first side of the structure and a second laser disposed on a second side of the structure opposite the first side.

3. The method of claim 2, wherein illuminating a non-specular ambient surface with the at least one coherent light source, thereby generating a first speckle pattern at the first 2D optical sensor array and generating a second speckle pattern at the 2D optical sensor array, comprises: projecting light from the first laser onto the non-specular ambient surface; receiving light reflected from the non-specular ambient surface using the first 2D optical sensor array to form the first speckle pattern; projecting light from the second laser onto the non-specular ambient surface; as well as Light reflected from the non-specular ambient surface is received using the second 2D optical sensor array to form the second speckle pattern.

4. The method according to claim 2, further comprising: providing a first aperture stop disposed in front of the first 2D optical sensor array, wherein the first aperture stop operates to exclude a range of emission angles of the second laser; as well as A second aperture stop is provided disposed in front of the second 2D optical sensor array, wherein the second aperture stop is operative to exclude a range of emission angles of the first laser.

5. The method according to claim 1, further comprising: providing a first imaging lens disposed in front of the first 2D optical sensor array, wherein the first imaging lens operates to magnify or reduce the first speckle pattern; as well as A second imaging lens is provided disposed in front of the second 2D optical sensor array, wherein the second imaging lens operates to magnify or reduce the second speckle pattern.

6. The method of claim 1, further comprising providing a collimating lens in front of the at least one coherent light source, the light being transmitted through the collimating lens.

7. The method of claim 6, further comprising providing a diffuser optically coupled to the collimating lens.

8. The method according to claim 7, wherein: The light from the at least one coherent light source coupled through the collimating lens and the diffuser is incident on a surface patch.

9. The method of claim 8, wherein the surface patch is a wall, ceiling, floor or furniture.

10. The method according to claim 1, further comprising: An inertial measurement unit (IMU) is provided to improve the accuracy of sensed translation and rotation.

11. An apparatus for sensing and distinguishing translational motion of a structure along a set of three independent axes and rotation of the structure about one of the set of three independent axes, the apparatus comprising: at least one coherent light source configured to emit light within a predetermined solid angle range and coupled to the structure; a first 2D optical sensor array coupled to the structure, the first 2D optical sensor array having a first normal vector pointing in a first direction; a second 2D optical sensor array coupled to the structure, the second 2D optical sensor array having a second normal vector pointing in a second direction, wherein the first normal vector and the second normal vector define a plane, and wherein the first normal vector is angled relative to the second normal vector in the plane; wherein the at least one coherent light source is configured to illuminate a non-specular ambient surface, thereby generating a first speckle pattern at the first 2D optical sensor array and generating a second speckle pattern at the 2D optical sensor array; The processing unit is configured as follows: determining a translation of the structure along a first of the set of three independent axes based on a first optical flow of the first speckle pattern on the first 2D optical sensor array and a second optical flow of the second speckle pattern on the second 2D optical sensor array, the translation comprising a non-zero projection on the plane between the first normal vector and the second normal vector, wherein the first optical flow and the second optical flow have opposite direction projections on a difference vector between the first normal vector and the second normal vector; determining a translation of the structure along a second of the set of three independent axes based on a third optical flow of the first speckle pattern on the first 2D optical sensor array and a fourth optical flow of the second speckle pattern on the second 2D optical sensor array, the translation comprising a non-zero projection on the plane outside an angular range between the first normal vector and the second normal vector, wherein the third optical flow and the fourth optical flow have a common directional projection on the difference vector between the first normal vector and the second normal vector; determining a translation of the structure along a third of the set of three independent axes including a non-zero component perpendicular to the plane based on the same sensed perpendicular optical flow of the first speckle pattern on the first 2D optical sensor array and the second speckle pattern on the second 2D optical sensor array; and A rotation of the structure about the first of the three independent axes is determined based on opposite perpendicular optical flows of the first speckle pattern on the first 2D optical sensor array and the second speckle pattern on the second 2D optical sensor array.

12. The apparatus of claim 11, wherein the at least one coherent light source comprises a first laser disposed on a first side of the structure and a second laser disposed on a second side of the structure opposite the first side.

13. The apparatus of claim 12, wherein the at least one coherent light source is configured to illuminate the non-specular ambient surface by: projecting light from the first laser onto the non-specular ambient surface; receiving light reflected from the non-specular ambient surface using the first 2D optical sensor array to form the first speckle pattern; projecting light from the second laser onto the non-specular ambient surface; as well as Light reflected from the non-specular ambient surface is received using the second 2D optical sensor array to form the second speckle pattern.

14. The apparatus according to claim 12, further comprising: A first aperture stop is disposed in front of the first 2D optical sensor array, wherein the a first aperture stop operative to exclude a range of emission angles of the second laser; as well as A second aperture stop is disposed in front of the second 2D optical sensor array, wherein the second aperture stop operates to exclude an emission angle range of the first laser.

15. The apparatus according to claim 11, further comprising: a first imaging lens disposed in front of the first 2D optical sensor array, wherein the first imaging lens operates to magnify or reduce the first spot pattern; as well as A second imaging lens is disposed in front of the second 2D optical sensor array, wherein the second imaging lens is operated to magnify or reduce the second spot pattern.

16. The apparatus of claim 11, further comprising a collimating lens located in front of the at least one coherent light source, wherein the light is transmitted through the collimating lens.

17. The apparatus of claim 16, further comprising a diffuser optically coupled to the collimating lens.

18. The device according to claim 17, wherein: The light from the at least one coherent light source is coupled through the collimating lens and the diffuser before being incident on the surface patch.

19. The device of claim 18, wherein the surface patch is a wall, ceiling, floor or furniture.

20. The apparatus of claim 11, further comprising an inertial measurement unit (IMU) configured to improve the accuracy of sensed translation and rotation.

Images