System and method for human-to-virtual object interaction

Abstract

A system for human interaction with a virtual object, comprising: a touch-sensitive surface configured to detect a location of a contact made on the touch-sensitive surface; a reference layer rigidly attached to the touch-sensitive surface and including one or more patterns; a display device configured to display a virtual object recorded in reference coordinates fixed with respect to the touch-sensitive surface; one or more image sensors rigidly attached to the display device, configured to capture an image of at least a portion of the one or more patterns; and at least one processor configured to determine a position and orientation of the display device relative to the touch-sensitive surface based on the captured image, and identify an interaction with the virtual object based on the detected position of the contact made on the touch-sensitive surface.

Description

[0001] This patent application is a divisional application of application number 201980064378.9 (PCT / US2019 / 044939) filed on August 2, 2019 and entitled “System and method for human-virtual object interaction”.

[0002] Related Applications

[0003] This application claims priority to U.S. Provisional Application No. 62 / 713,887, filed on August 2, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present disclosure generally relates to the fields of augmented reality and virtual reality. In particular, the present disclosure relates to the interaction between people and virtual objects. Background Art

[0005] The interaction between users and virtual objects is a hot topic that is widely studied in the current field of augmented reality and virtual reality. Commonly used interaction methods can generally be divided into direct interaction methods and indirect interaction methods. Direct interaction methods track the user's gestures and postures or the spatial position of the handheld device so that the handheld device can be accurately recorded in the virtual space, and the virtual space itself can be accurately overlaid or displayed on top of the real world. As a result, the user can directly interact with the virtual object at its display position. Indirect interaction methods usually include an indicator that interacts with the virtual elements, and the user controls the movement and action of the indicator via an input device. In addition, the movement of the indicator can be scaled or offset relative to the recorded input movement, so that for a small interaction area, the user can interact with virtual objects that are far away or distributed in a large space.

[0006] Direct interaction methods can employ technologies such as hand tracking, body tracking, and six-degree-of-freedom handheld controllers. The hand tracking method typically has a gesture capture and recognition system, where a gesture can be captured by an image acquisition sensor or a glove equipped with orientation and bending sensors to reconstruct the gesture. Then, similar to the case of a hand interacting with a real object in the physical world, the reconstructed gesture or hand and its spatial position and orientation can be used to simulate interaction with a virtual object. Similarly, a six-degree-of-freedom handheld controller has its translation and orientation tracking so that it can be recorded in a virtual space and used to create interactions with a virtual object at its display location.

[0007] Indirect interaction methods can use variations of traditional input devices (such as touchpads, mice, or keyboards). Users can interact with these devices to control the movement of pointers in virtual space and interact with virtual elements. For example, an interactive system using a mouse or touchpad can record the user's hand movements or finger movements on a table or touchpad surface, so that a virtual pointer moves according to the recorded movement and is used to apply actions to virtual elements. The movement of the virtual pointer can be scaled, and its position can be offset relative to the position of the hand or finger, so that the user can navigate the pointer throughout the virtual space without running out of the space in the interactive area of ​​the input device. In addition, sometimes the input device only records relative movement, so that the user can further expand the range of movement of the virtual pointer by performing multiple relative movements on the input device.

[0008] While direct interaction using, for example, hand tracking or handheld controllers provides a realistic experience and a high degree of freedom in controlling virtual objects, the need to constantly hold the hands in the air and move them around in the physical world can quickly introduce fatigue and prevent prolonged use. In addition, it is challenging to implement reliable and high-fidelity spatial tracking of hands or bodies with relatively low-cost sensors and a streamlined user setup experience. Indirect interaction using, for example, a keyboard, mouse, or touchpad system provides a familiar experience from the personal computer / smartphone era and allows users to easily interact with virtual content, but it reduces the sense of immersion created by augmented reality or virtual reality systems. Summary of the invention

[0009] Methods and systems are disclosed that allow users to view and directly interact with three-dimensional (3D) and two-dimensional (2D) virtual content accurately overlaid on an interactive area of ​​a touch input device. Such a system maintains the immersion of an augmented reality or virtual reality system by allowing the user to directly interact with virtual objects at their display location and interact with the precise input movements recorded. At the same time, the integration of the touch input system provides a physical surface for interaction and hand resting. As a result, accurate, responsive, reliable tracking of input movements and an effortless interaction experience can be achieved.

[0010] In one aspect of the present disclosure, a system for human interaction with virtual objects comprises: a touch-sensitive surface configured to detect the location of contacts made on the touch-sensitive surface; a reference layer rigidly attached to the touch-sensitive surface and comprising one or more patterns; a display device configured to display a virtual object recorded in reference coordinates fixed relative to the touch-sensitive surface; one or more image sensors rigidly attached to the display device and configured to capture an image of at least a portion of the one or more patterns; and at least one processor configured to determine a position and orientation of the display device relative to the touch-sensitive surface based on the captured image, and to identify an interaction with the virtual object based on the detected location of the contacts made on the touch-sensitive surface.

[0011] In one aspect of the present disclosure, the virtual object may be a three-dimensional virtual object. In another aspect of the present disclosure, the virtual object may be a two-dimensional virtual element. In one aspect of the present disclosure, the display device is a perspective display device.

[0012] In one aspect of the disclosure, the one or more patterns include one or more fiducial markers. In one aspect of the disclosure, wherein the one or more fiducial markers are configured to absorb infrared light and the one or more image sensors are configured to sense infrared light. In one aspect of the disclosure, each of the one or more fiducial markers includes a rectangle containing an internal grid representation of a binary code. In one aspect of the disclosure, each of the one or more fiducial markers includes a plurality of image features with known positions, wherein each image feature corresponds to a unique feature descriptor.

[0013] In one aspect of the disclosure, one or more patterns include multiple light sources with known positions. In one aspect of the disclosure, the multiple light sources are infrared light sources, and the one or more image sensors are configured to sense infrared light. In one aspect of the disclosure, the multiple light sources are configured to be turned on in a predetermined order.

[0014] In one aspect of the present disclosure, the one or more patterns include a mask and one or more light sources, wherein at least a portion of the light emitted from the one or more light sources and passing through the mask is captured by the one or more image sensors. In one aspect of the present disclosure, the one or more patterns further include a diffuser configured to diffuse the light emitted from the one or more light sources. In one aspect of the present disclosure, the one or more patterns further include a light guide plate configured to receive light emitted from at least one side of the light guide plate by the one or more light sources and guide at least a portion of the light to the mask above the light guide plate.

[0015] In one aspect of the disclosure, the touch-sensitive surface is at least partially transparent and the reference layer is arranged below the touch-sensitive surface. In one aspect of the disclosure, the reference layer is arranged adjacent to at least one side of the touch-sensitive surface. In one aspect of the disclosure, the reference layer is arranged above the touch-sensitive surface.

[0016] In one aspect of the disclosure, at least one processor is configured to: identify an interaction with a virtual object when the detected location of the contact matches the location of the virtual object. In one aspect of the disclosure, the virtual object is lifted from the touch-sensitive surface, and at least one processor is configured to: identify an interaction with the virtual object when the detected location of the contact matches the location of a virtual footprint projected from the virtual object onto the touch-sensitive surface. In one aspect of the disclosure, when the interaction is identified, the display device displays a virtual two-dimensional menu that approximates the virtual footprint.

[0017] In one aspect of the present disclosure, a system for human interaction with virtual objects includes: a touch-sensitive surface configured to detect the location of contacts made on the touch-sensitive surface; a display device configured to display virtual objects; one or more ultrasonic transmitters rigidly attached to one of the touch-sensitive surface and the display device, configured to transmit ultrasonic signals; one or more ultrasonic receivers rigidly attached to the other of the touch-sensitive surface and the display device, configured to receive ultrasonic signals transmitted by the ultrasonic transmitters; and at least one processor configured to determine a position and orientation of the display device relative to the touch-sensitive surface based at least on a time of flight of the received ultrasonic signals, and to identify interaction with the virtual object based on the detected location of the contacts.

[0018] In one aspect of the disclosure, the system further includes an inertial measurement unit rigidly attached to the touch-sensitive surface and / or an inertial measurement unit rigidly attached to the display device.

[0019] In one aspect of the present disclosure, a method for human interaction with a virtual object includes: detecting the location of a contact made on a touch-sensitive surface; displaying, using a display device, a virtual object recorded in reference coordinates fixed relative to the touch-sensitive surface; capturing, using one or more image sensors rigidly attached to the display device, an image of at least a portion of one or more patterns on a reference layer rigidly attached to the touch-sensitive surface; determining, based on the captured image, a position and orientation of the display device relative to the touch-sensitive surface; and identifying an interaction with the virtual object based on the detected location of the contact.

[0020] In one aspect of the disclosure, the one or more patterns include one or more fiducial markers. In one aspect of the disclosure, the one or more patterns include multiple light sources with known positions. In one aspect of the disclosure, the one or more patterns include a mask and one or more light sources, wherein at least a portion of light emitted from the one or more light sources and passing through the mask is captured by the one or more image sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A structural diagram of an exemplary user interaction system according to one aspect of the present disclosure is shown.

[0022] Figure 2 A structural diagram of a first exemplary touch input device according to one aspect of the present disclosure is shown.

[0023] Figure 3 A structural diagram of a second exemplary touch input device according to one aspect of the present disclosure is shown.

[0024] Figure 4 A structural diagram of a third exemplary touch input device according to one aspect of the present disclosure is shown.

[0025] Figure 5A A first exemplary fiducial marker according to one aspect of the present disclosure is shown.

[0026] Figure 5B A second exemplary fiducial marker according to one aspect of the present disclosure is shown.

[0027] Figure 5C A third exemplary fiducial marker according to one aspect of the present disclosure is shown.

[0028] Figure 6 A cross-sectional view of an exemplary reference layer is shown.

[0029] Figure 7 A cross-sectional view of another exemplary reference layer is shown.

[0030] Figure 8 A cross-sectional view of a third exemplary reference layer is shown. DETAILED DESCRIPTION

[0031] Figure 1A block diagram of an exemplary system 100 that allows a user to interact with a virtual object according to one aspect of the present disclosure is shown. The system 100 may include: a touch input device 101, which includes a reference layer 102 and an interactive surface 103; a display device 105; a posture tracking device 106; and a computing unit 107, which is operably connected to the touch input device 101, the display device 105, and the posture tracking device 106 wirelessly or via one or more cables. In some embodiments, the touch input device 101 may include a touch pad that includes a tactile sensor, wherein the interactive surface 103 may be an active surface of the tactile sensor that detects contact of a finger or an object (e.g., a stylus) with the surface. In some embodiments, the reference layer 102 may include a predetermined set of reference patterns. In some embodiments, the reference layer 102 may include a plurality of light sources, such as light emitting diodes (LEDs), arranged in a predetermined pattern. In some embodiments, an exemplary virtual object 108 is recorded in a reference coordinate 104 that is fixed relative to the touch input device 101, and the relative position and orientation between the reference coordinate 104 and the interactive surface 103 is known, so that the position and movement of the detected touch input can be directly mapped to the virtual space. In some embodiments, the posture tracking device 106 can include a single or multiple image sensors rigidly attached to the display device 105. Multiple image sensors can be advantageous in situations where a wider field of view is required or where an object such as a user's hand blocks a portion of the field of view. In some embodiments, each image sensor can also include a filter that allows light having a predetermined range of wavelengths to pass while attenuating the intensity of light having other wavelengths.

[0032] In some embodiments, display device 105 can be a perspective display device, through which the viewer can perceive the virtual content generated by the computer and the real world. As a result, system 100 can be used in augmented reality applications. In some embodiments, display device 105 can be opaque, so that it can block light from the real world and only display the virtual content generated by the computer. As a result, system 100 can be used in virtual reality applications. In some embodiments, display device 105 can be a head-mounted device, which is placed in front of the viewer's eyes 109 when in use.

[0033] In some embodiments, the interactive surface 103 is capable of detecting and reporting the precise location of a touch event on the surface, wherein the touch event may be generated by contact between an object and the surface, wherein the object may be a user's finger or hand, or a stylus, etc. In some embodiments, the touch input device 101 may also detect and report the shape of the contact area of ​​the touch event. In some embodiments, the touch input device 101 may further detect and report the force distribution on the contact area of ​​the touch event.

[0034] In some embodiments, the reference layer 102 can be sensed by the posture tracking device 106 to determine the position and orientation of the touch input device 101 relative to the posture tracking device 106. In some embodiments, the reference layer 102 can be a reference pattern layer, which can include a predetermined set of points, lines or shapes. In some embodiments, the reference layer 102 can include a light emitting diode layer arranged in a predetermined pattern.

[0035] In some embodiments, the interactive surface 103 may include a tactile sensor that accurately measures the location of contact between the sensor and a finger or object. In some embodiments, the interactive surface 103 may be completely transparent or translucent. As a result, the reference layer 102 may be disposed below the interactive surface 103 while still being able to be sensed by the posture tracking device 106. In some embodiments, the interactive surface 103 may be opaque. As a result, the reference layer 102 may be disposed on top of the interactive surface 103 or attached to one or more sides of the interactive surface 103. In some embodiments, the tactile sensor may further measure the area and / or force distribution of the contact. In some embodiments, the tactile sensor may be resistive sensing or capacitive sensing. The tactile sensor may be of any type that a person skilled in the art deems suitable for performing the functions described herein.

[0036] As described above, the touch input device 101 may include a reference layer 102. In some embodiments, the reference layer 102 may include a predetermined set of reference patterns, wherein the reference patterns include a predetermined combination of features, including shapes, lines, and points, wherein the size, position, or orientation of these features is known. As a result, when a portion or all of the reference pattern is captured by one or more image sensors, the position and orientation of the pattern can be determined. In some embodiments, the reference pattern can be printed or etched on a supporting substrate layer, for example, using a material that absorbs visible light and / or infrared light. In some embodiments, the reference pattern can be created by applying an opaque mask over a diffuse illumination source and partially cutting it off. The reference pattern can be manufactured in many forms that a person skilled in the art would deem suitable for performing the functions described herein.

[0037] Figure 5 Figure 6 and Figure 7 Non-limiting examples of fiducial markers that can be used with embodiments of the present disclosure are shown.

[0038] In some embodiments, the reference layer may include multiple light sources, such as light emitting diodes (LEDs), arranged in a predetermined pattern, where the position of each LED is known. As a result, when a portion or all of the LEDs are captured by one or more image sensors, the position and orientation of the pattern can be determined. In some embodiments, the LEDs are lit sequentially so that: in each frame, only one or a few LEDs are captured by the image sensor. Because if all LEDs are lit at the same time, multiple LEDs may share similar characteristics in the captured image frame, and this may cause uncertainty problems when a correspondence needs to be established between each observed LED and each known position. Therefore, the uncertainty problem can be solved by turning on the LEDs in a predetermined order. At the same time, because not all LEDs need to be turned on all the time in order for the posture tracking device to determine the position and orientation, lighting the LEDs sequentially can save precious battery power.

[0039] In some embodiments, the posture tracking device 106 may include a single or multiple image sensors. In some embodiments, each image sensor may also contain a filter that only allows light having a predetermined wavelength range (e.g., infrared light) to pass through while attenuating the intensity of light having other wavelengths (e.g., visible light), wherein the predetermined range may depend on the wavelength of light reflected or emitted by the above-mentioned pattern. As a result, the pattern can be clearly captured by the image sensor, while other features in the sensor's field of view may be partially or completely invisible to the sensor. In some embodiments, the posture tracking device 106 may further include a lighting device, wherein the lighting device may include a single or multiple light emitting diodes.

[0040] The computing unit 107 may include one or more processors. Figure 1In the example shown, the computing unit 107 is incorporated into a head mounted device including a posture tracking device 106 and a display device 105, but the computing unit 107 can be arranged in a separate computer or incorporated into the touch input device 101 instead. The computing unit 107 can generate a virtual space including virtual content, wherein the virtual content can include three-dimensional virtual objects and / or two-dimensional virtual elements. The computing unit 107 can also generate information about the virtual content, wherein the information can include spatial position, orientation, shape, kinematics, dynamics, and graphics rendering characteristics. When a touch event is detected, the touch position reported by the touch input device 101 and the position of the virtual object overlaid on the top of the interactive surface 103 will allow the computing unit 107 to determine which virtual object is being interacted with. In addition, the type of touch event and the characteristics of the virtual object can define different interactions and responses. At the same time, the posture tracking device 106 provides the computing unit 107 with the necessary measurement data, so that the relative spatial relationship between the display device 105 and the touch input device 101 can be determined. The relative spatial relationship generally refers to the relative translation and rotation between two entities. Therefore, virtual objects and responses triggered by user interactions can be rendered and displayed appropriately from the viewer's perspective.

[0041] For example, the virtual object can be enlarged while being interacted with, and text information associated with the particular object can be optionally displayed. As another example, the virtual object can be dragged from a first position to a second position through a touch interaction.

[0042] In one embodiment, the fiducial pattern may include a plurality of square-based fiducial markers, each of which has an outer boundary and an inner grid representation of a binary code. Examples of such fiducial markers are shown in FIG. Figure 5A As shown in . When captured by the image sensor, the fiducial markers in the captured frame are extracted by discarding any shapes that are not 4-vertex polygons, do not have valid binary codes, or do not meet the size threshold. The corners of each valid fiducial marker are detected by a corner detection algorithm. With the known position of the corners of the fiducial marker, the relative position and orientation of the marker with respect to the image sensor can be estimated, for example, by solving the perspective n-point problem. Therefore, pose estimation based on the fiducial pattern is achieved.

[0043] In another embodiment, the reference pattern may include a predetermined image object containing a plurality of features with known positions in the image object and known descriptors. Features in computer vision or image processing are different local structures found in an image, such as "edges" (a group of points in an image with strong gradient magnitudes), "corners / interest points" (a group of points where the direction of the gradient changes rapidly within a local area), or local image patches. Descriptors encode the characteristics of a feature, for example, the magnitude and orientation of a local gradient of pixel intensity, a vector of intensity comparisons between a group of pixel pairs around the feature. The descriptor may be in a variety of forms, including numerical values, vectors of numerical values, or vectors of Boolean variables. Descriptors can be used to uniquely identify corresponding features in an image. For example, in the case of a BRIEF (Binary Robust Independent Elementary Features) descriptor, the Hamming distance between a known descriptor and the descriptor of a candidate feature is calculated, and a match is confirmed if the distance is less than a threshold.

[0044] When an image target is captured by an image sensor, a feature detection algorithm is used to extract all candidate feature points within the captured frame, and corresponding descriptors are calculated for the candidate feature points. By comparing the descriptors, some candidate feature points are matched with known feature points in the image target. For example, by solving the perspective n-point problem, the matched pairs are used to estimate the relative position and orientation of the image target with respect to the image sensor. Thus, pose estimation based on the reference pattern is achieved.

[0045] Figure 2 is a structural diagram of a first exemplary touch input device. Figure 2 In the illustrated embodiment, the interactive layer 202 is disposed on top of the reference layer 201, wherein the interactive layer 202 is fully or partially transparent. The interactive layer 202 may include a tactile sensor as described above. The reference layer 201 may include a predetermined set of reference patterns as described above. Alternatively, the reference layer 201 may include a plurality of light emitting diodes (LEDs) arranged in a predetermined pattern as described above.

[0046] Figure 3 is a structural diagram of a second exemplary touch input device. Figure 3 In the illustrated embodiment, the reference layer 301 is rigidly attached to one side of the interactive layer 302, wherein the interactive layer 302 may be opaque. The interactive layer 302 may include a tactile sensor as described above. The reference layer 301 may include a predetermined set of reference patterns as described above. Alternatively, the reference layer may include a plurality of light emitting diodes (LEDs) arranged in a predetermined pattern as described above.

[0047] As an optional example, Figure 1 As shown, the reference layer 102 may be arranged around four sides of a rectangular interaction surface 103 .

[0048] In some embodiments, the touch input device may include multiple ultrasonic transmitters placed in a predetermined pattern, and the posture tracking device may include multiple ultrasonic receivers. In some embodiments, the posture tracking device may include multiple ultrasonic transmitters placed in a predetermined pattern, and the touch input device may include multiple ultrasonic receivers. As a result, the distance between the transmitter and the receiver can be determined by measuring the propagation time of the ultrasonic signal. Therefore, the position and orientation of the touch input device can be determined.

[0049] Figure 4 is a structural diagram of a third exemplary touch input device using an ultrasonic transmitter or receiver. Figure 4 In the illustrated embodiment, a support frame 402 is attached to four sides of an interactive layer 401, and four ultrasonic transmitters 403, 404, 405, 406 are disposed on top of the support frame 402 and distributed at four corners. The interactive layer 402 may include a tactile sensor as described above. Alternatively, four ultrasonic receivers 403, 404, 405, 406 may be disposed on top of the support frame 402.

[0050] Various methods can be used to track the position and orientation of a virtual object using ultrasonic receivers and transmitters. For example, in one embodiment, three ultrasonic receivers are rigidly attached to one of a display device and a touch input device in a non-collinear arrangement, three transmitters are rigidly attached to the other of the display device and the touch input device, and a computing unit is coupled to the transmitter and the receiver. The three transmitters generate ultrasonic pulses of three different frequencies, respectively. Each of the three receivers separates the received ultrasonic waves with three different frequencies into three signals, resulting in a total of nine signals. Based on the time-of-flight principle, the nine signals are processed into nine distances between each of the three transmitters and each of the three receivers. Therefore, the relative orientation and position between the transmitter assembly and the receiver assembly can be estimated.

[0051] In another embodiment, an ultrasonic transmitter and a 9-axis inertial measurement unit (IMU) are rigidly attached to one of the display device and the touch input device, and three ultrasonic receivers and a 9-axis IMU are rigidly attached to the other of the display device and the touch input device, wherein the receiver is arranged in a non-collinear arrangement, and a computing unit is coupled to the transmitter, the receiver, and the IMU. Alternatively, three transmitters and one receiver can be used. The transmitter generates ultrasonic sound pulses of known frequency, and the receiver converts the received ultrasonic pulses into three signals. Based on the time of flight principle, the signal generates three distances between the transmitter and the three receivers, respectively. As a result, the relative position between the transmitter and the receiver assembly can be calculated. These IMUs measure the absolute orientation of the display device and the touch input device so that the relative orientation between them can be determined.

[0052] In some embodiments, a footprint may be displayed for the lifted virtual object. Because the user interaction is sensed by a touch input device, the interaction is limited to the vicinity of a 2D plane. However, the virtual content may be displayed above the touch input device at a non-negligible vertical distance. To overcome this limitation, in one embodiment, a virtual footprint projected from the lifted virtual object onto the interactive layer is displayed by a display device. As a result, the user may interact with the lifted virtual object via its virtual footprint using various touch gestures. For example, the user may perform a pinch gesture on the touch input device on the area of ​​the virtual footprint to scale the corresponding virtual object, or the user may perform a press and drag gesture on the virtual footprint to move the corresponding virtual object. In addition, when the user touches the area of ​​the virtual footprint, a virtual two-dimensional menu element may be displayed on the interactive layer near the virtual footprint to provide additional operations on the corresponding virtual object. The user may click on different areas of the interactive layer displaying menu items to activate related functions. For example, additional operations may include, but are not limited to: starting an animation associated with the corresponding virtual object, deleting the corresponding virtual object, or changing the properties of the corresponding virtual object.

[0053] Figure 6 is a cross-sectional view of an exemplary reference layer. Figure 6 In the illustrated embodiment, the mask layer 603 is arranged above the light source layer 601, and an optional light diffuser 602 can be located between the light source layer 601 and the mask layer 603. The mask layer 603 includes a light-transmitting portion illustrated as a bright block and a light-shielding portion illustrated as a dark block. Light emitted by the light source in the light source layer 601 can be diffused by the light diffuser 602 and pass through the light-transmitting portion of the mask layer 603 to form one or more of the above patterns when captured by the image sensor. In some embodiments, the light from the light source can be infrared light, and the light-transmitting portion and the light-shielding portion respond to infrared wavelengths.

[0054] In some embodiments, the light shielding portion may include a polymer with a light shielding additive. In some embodiments, the light shielding portion may include a light shielding paint deposited on a substrate. In some embodiments, the light-transmitting portion is simply formed by a gap, or lacks any material. The material of the mask layer 603 can be any type that a person skilled in the art considers suitable for performing the functions described herein.

[0055] In some embodiments, the interaction surface is located above the mask layer 603. In some embodiments, the interaction surface is located below the mask layer 603. In some embodiments, the light shielding material is directly deposited on the interaction surface to form the mask layer 603.

[0056] Figure 7 is a cross-sectional view of another exemplary reference layer. Figure 7 In the illustrated embodiment, mask layer 703 is disposed above light guide plate 701 and may have the same configuration as mask layer 603 described above. Light source 702 is disposed to emit light laterally toward light guide plate 701 from at least one side of the light guide plate. Light emitted from light source 702 is redirected by light guide plate 701 to mask layer 703 and passes through mask layer 703 to form one or more patterns described above when captured by an image sensor. In some embodiments, light guide plate 701 may include an array of microlenses on a bottom surface, and light from light source 702 is reflected internally along the length of light guide plate 701 (e.g., at Figure 7 The light from the light source can be infrared light, and the light-transmitting and light-shielding portions of the mask layer 703 are responsive to infrared wavelengths.

[0057] In some embodiments, the interaction surface is located above the mask layer 703. In some embodiments, the interaction surface is located below the mask layer 703. In some embodiments, the light shielding material is deposited directly on the touch-sensitive surface to form the mask layer 703.

[0058] Figure 8 is a cross-sectional view of a third exemplary reference layer. Figure 8 In the illustrated embodiment, the light guide plate 801 is located above a reference layer 803 that includes one or more reference patterns. The light source 802 is arranged to emit light laterally to the light guide plate 801 from at least one side of the light guide plate 801. The light emitted by the light source is redirected by the front light guide plate to illuminate the reference layer 803 and form the one or more reference patterns described above in the image frame when captured by the image sensor. In some embodiments, the light guide plate 801 can include a microlens array on the top surface, and the light from the light source is reflected internally along the length of the light guide plate 801 (for example, at Figure 8The light guide plate 801 may be any type deemed suitable by one skilled in the art to perform the functions described herein. In some embodiments, the light from the light source may be infrared light, and the reference pattern may be responsive to infrared wavelengths.

[0059] In some embodiments, the interaction surface is located above the light guide plate 801. In some embodiments, the interaction surface is located below the reference layer 803. In some embodiments, the interaction surface is located between the light guide plate 801 and the reference layer 803.

[0060] For the purpose of illustration, the foregoing description has been described with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. These embodiments are selected and described in order to best explain the principles of the invention and its practical application, so that others skilled in the art can best utilize the invention and various embodiments with various modifications suitable for the specific use contemplated.

Claims

1. A system for human-virtual object interaction, comprising: a touch-sensitive surface configured to detect a location of a contact made on the touch-sensitive surface; a reference layer rigidly attached to at least one side of the touch-sensitive surface, wherein The reference layer comprises: Light guide plate; a reference layer comprising one or more patterns; and a light source positioned such that light emitted from the light source enters the light guide plate via a lateral side of the light guide plate, is reflected internally along the length of the light guide plate, and is redirected to illuminate the reference layer and form the one or more patterns in the image frame; a wearable display device configured to display a virtual object recorded in reference coordinates fixed relative to the touch-sensitive surface; one or more image sensors rigidly attached to the display device and configured to capture an image of at least a portion of the one or more patterns in the reference layer; and at least one processor configured to determine a position and orientation of the display device relative to the touch-sensitive surface based at least in part on image data associated with one or more patterns found in the captured images, acquiring touch data from the touch-sensitive surface, the touch data comprising locations of detected contacts made on the touch-sensitive surface; causing a virtual object to be displayed by the wearable display device; generating an operation on a virtual object based on the detected location of the contact made on the touch-sensitive surface; The operation of the virtual object is displayed by the wearable display device.

2. The system according to claim 1, wherein: The one or more patterns include a light-transmitting portion and a light-shielding portion.

3. The system according to claim 1, wherein: The one or more patterns include light absorbing portions.

4. The system according to claim 1, wherein: The virtual object is a two-dimensional virtual object.

5. The system according to claim 1, wherein: The virtual object is a three-dimensional virtual object.

6. The system according to claim 1, wherein: The display device is a see-through display device.

7. The system according to claim 1, wherein: The one or more patterns include one or more fiducial marks.

8. The system according to claim 7, wherein: The one or more fiducial markers are configured to absorb infrared light, and the one or more image sensors are configured to sense infrared light.

9. The system according to claim 7, wherein: Each of the one or more fiducial markers comprises a rectangle containing an internal grid representation of a binary code.

10. The system according to claim 7, wherein: Each of the one or more fiducial markers comprises a plurality of image features having known positions, wherein each of the image features corresponds to a unique feature descriptor.

11. The system according to claim 1, wherein: A planar light source layer is arranged below the reference layer, and a light-transmitting portion and a light-shielding portion of the reference layer are configured as a mask.

12. The system of claim 1, wherein: A planar light source layer is arranged between the touch-sensitive surface and the reference layer.

13. A method for human-virtual object interaction, comprising: detecting a location of a contact made on a touch-sensitive surface; displaying, with a display device, a virtual object recorded in reference coordinates fixed relative to the touch-sensitive surface; emitting light into a lateral side of the light guide with a light source to illuminate one or more patterns of a reference layer rigidly attached to at least one side of the touch-sensitive surface; capturing an image of at least a portion of the one or more patterns using one or more image sensors rigidly attached to the display device; determining a position and orientation of the display device relative to the touch-sensitive surface based on the captured image; acquiring touch data from the touch-sensitive surface, the touch data comprising locations of detected contacts made on the touch-sensitive surface; causing a virtual object to be displayed by the wearable display device; generating an operation on a virtual object based on the detected location of the contact made on the touch-sensitive surface; The operation of the virtual object is displayed by the wearable display device.

14. The method according to claim 13, wherein: The one or more patterns of the reference layer include a light-transmitting portion and a light-shielding portion, and the method further includes: using one or more patterns of the reference layer to allow light to pass through the light-transmitting portion; and The light is shielded by the light shielding portion using one or more patterns of the reference layer.

15. The method according to claim 13, wherein: The one or more patterns of the reference layer include a light-transmitting portion, and the method further includes: using the one or more patterns of the reference layer so that light is absorbed by the light-shielding portion.

16. The method according to claim 13, wherein: The one or more patterns include one or more fiducial marks.

17. The method according to claim 13, wherein: The one or more patterns include a plurality of light sources having known positions.

18. The method according to claim 13, wherein: The one or more patterns include a mask and a light source, wherein at least a portion of light emitted from the light source and passing through the mask is captured by the one or more image sensors.

Images