Method and apparatus for providing input for a head-mounted image display device - Patents.com

Abstract

To provide a suitable method of and a suitable apparatus for providing an input for a head mount type image display device.SOLUTION: The present invention is directed to an apparatus used jointly with an image display device arranged so as to be mounted to a head of a user. The apparatus has a screen, and a processing unit arranged so as to allocate a first area of the screen to detect a finger action of the user. The processing unit, based on the finger action of the user detected by the first area allocated in the screen of the apparatus, generates an electronic signal for causing change of a content displayed on the display device.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to head-mounted image display devices and methods and apparatus for providing input for such image display devices. [Background technology]

[0002] Modern computing and display technologies have facilitated the development of so-called "mixed reality" (MR) systems for "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images or parts thereof are presented to a user in a manner that appears or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs. AR scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user (i.e., transparency to real-world visual inputs). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to real-world visual inputs.

[0003] MR systems can generate and display color data, which increases the realism of the MR scenario. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Projecting the color sub-images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) can result in a smooth color MR scenario in the user's memory.

[0004] Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility Application No. 14 / 555,585, filed November 27, 2014 (Attorney Docket No. ML.20011.00), the contents of which are expressly and fully incorporated herein by reference in their entirety as if fully set forth.

[0005] The MR system may employ at least a wearable display device (e.g., a head-mounted display, a helmet-mounted display, or smart glasses) that is loosely coupled to the user's head and thus moves as the user's head moves. When the user's head movement is detected by the display device, the data being displayed may be updated (e.g., "warped") to account for changes in head pose (i.e., the orientation and / or location of the user's head).

[0006] As an example, when a user wearing a head mounted display device views a virtual representation of a virtual object on the display and walks around the area in which the virtual object appears, the virtual object can be rendered per viewpoint to give the user the perception of walking around objects that occupy real space. When a head mounted display device is used to present multiple virtual objects, head pose measurements can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion.

[0007] AR-enabling head-mounted display devices provide simultaneous viewing of both real and virtual objects. With an "optical see-through" display, a user can look through a transparent (or semi-transparent) element in the display system and directly view light from real objects in the environment. The transparent element, often referred to as a "combiner," superimposes light from the display over the user's view of the real world, from which the display projects an image of the virtual content over a see-through view of the real objects in the environment. A camera may be mounted on the head-mounted display device to capture images or video of the scene being viewed by the user.

[0008] Current optical systems, such as those in MR systems, optically render virtual content. The content is "virtual" in that it does not correspond to actual physical objects located at discrete positions in space. Instead, the virtual content exists only in the brain (e.g., visual centers) of a user of a head-mounted display device when stimulated by light beams directed at the user's eyes.

[0009] Disclosed herein are methods and apparatus for providing input for a head-mounted image display device (e.g., an MR device, an AR device, a VR device, etc.). Summary of the Invention [Means for solving the problem]

[0010] The device for use with a visual display device configured to be head-worn by a user includes a screen and a processing unit configured to allocate a first area of ​​the screen to sense a user's finger action, the processing unit configured to generate an electronic signal to cause a change in content displayed by the visual display device based on the user's finger action sensed by the allocated first area of ​​the screen of the device. As used herein, the term "finger action" may include actions performed by one or more fingers, and may include actions such as swiping, pinching, unpinching, tapping, pressing, holding, twisting, changing direction, etc.

[0011] Optionally, the screen has a touch-sensitive area, and the allocated first area is less than a total area of ​​the touch-sensitive area.

[0012] Optionally, the allocated first area has a shape that corresponds to the shape of a screen of the image display device.

[0013] Optionally, the assigned first area has dimensions based on the brand and / or model of the device.

[0014] Optionally, the allocated first area has dimensions based on characteristics of the apparatus and characteristics of the image display device.

[0015] Optionally, the processing unit is configured to ignore input generated by the user using a portion of the touch-sensitive area that is not part of the assigned first area and that is not part of the assigned button.

[0016] Optionally, the processing unit is also configured to operate a feedback component in response to a user's finger action.

[0017] Optionally, the first area has a boundary and the user's finger action includes moving the user's finger to a location that crosses, reaches, or is within a predefined distance from the boundary.

[0018] Optionally, the first area has one or more boundaries at least partially surrounding the reference location, and the processing unit is configured to operate the feedback component in response to the user's finger reaching a predefined distance from the reference location.

[0019] Optionally, the processing unit is configured to operate the feedback component to generate different types of feedback based on different respective spatial relationships between one or more fingers of the user relative to the first area.

[0020] Optionally, the different types of feedback include a first tactile impulse with a first amplitude and a second tactile impulse with a second amplitude different from the first amplitude.

[0021] Optionally, the different types of feedback include a first number of tactile impulses and a second number of tactile impulses different from the first number.

[0022] Optionally, the different types of feedback include a first series of tactile impulses with a first frequency and a second series of tactile impulses with a second frequency different from the first frequency.

[0023] Optionally, the different respective spatial relationships include different distances between (1) one of the user's one or more fingers, or a point between two fingers, and (2) a reference location within the assigned first area.

[0024] Optionally, the reference location includes a centre of the assigned first area.

[0025] Optionally, the different distance exceeds a threshold.

[0026] Optionally, the different respective spatial relationships include one of the one or more fingers, or a point between two fingers, arriving at different respective distances from a boundary of the assigned first area.

[0027] Optionally, the assigned first area has a first boundary and a second boundary, and the different types of feedback include at least a first type of feedback and a second type of feedback, and the processing unit is configured to operate the feedback component and generate the first type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the first boundary, and the processing unit is configured to operate the feedback component and generate the second type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the second boundary.

[0028] Optionally, the first boundary includes a left or right boundary of the allocated first area, and the second boundary includes an upper or lower boundary.

[0029] Optionally, the processing unit is configured to operate the feedback component based on the swipe direction.

[0030] Optionally, the processing unit is configured to obtain an input signal associated with a pinch or unpinch action performed on the allocated first area of ​​the screen.

[0031] Optionally, the processing unit is configured to generate an electronic signal for varying a size of content displayed by the image display device based on an input signal associated with a pinch or unpinch action.

[0032] Optionally, the device further includes an orientation sensor for sensing an orientation of the device, and the processing unit is also configured to generate an electronic signal for changing content displayed by the image display device based on an input signal associated with the pinch or unpinch action and the sensed orientation of the device.

[0033] Optionally, the processing unit is configured to generate an electronic signal for contracting or expanding the content in a first plane when a pinch or unpinch action is sensed by the device while the device is in a first orientation, and the processing unit is configured to generate an electronic signal for contracting or expanding the content in a second plane when a pinch or unpinch action is sensed by the device while the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0034] Optionally, the device has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0035] Optionally, the device has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0036] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0037] Optionally, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0038] Optionally, the first plane is perpendicular to the second plane.

[0039] Optionally, the processing unit is configured to obtain an input signal associated with a swipe action performed on the allocated first area of ​​the screen.

[0040] Optionally, the processing unit is configured to generate an electronic signal for changing the content by moving the content in response to an input signal associated with a swipe action.

[0041] Optionally, the device further includes an orientation sensor for sensing an orientation of the device, and the processing unit is configured to generate an electronic signal for varying content displayed by the image display device based on the input signal associated with the swipe action and the sensed orientation of the device.

[0042] Optionally, the processing unit is configured to generate electronic signals for moving the content in a first plane when a swipe action is sensed by the device while the device is in a first orientation, and the processing unit is configured to generate electronic signals for moving the content in a second plane when a swipe action is sensed by the device while the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0043] Optionally, the device has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0044] Optionally, the device has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0045] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0046] Optionally, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0047] Optionally, the first plane is perpendicular to the second plane.

[0048] Optionally, the content is within the virtual three-dimensional environment and the processing unit is configured to generate electronic signals for altering the content displayed by the image display device by moving the content closer to or farther away from the user when a swipe action is sensed by the device while an orientation of the device is substantially parallel to a horizontal plane.

[0049] Optionally, the content is within a virtual three-dimensional environment and the processing unit is configured to generate electronic signals for moving content displayed by the image display device in a vertical plane within the three-dimensional environment when a swipe action is sensed by the device while an orientation of the device is substantially perpendicular to a horizontal plane.

[0050] Optionally, the device further comprises an orientation sensor for sensing an orientation of the device, the processing unit being configured to generate the electronic signal based on the sensed orientation of the device.

[0051] Optionally, the processing unit is configured to generate electronic signals to cause the content to expand in one or more directions based on the sensed orientation of the device.

[0052] Optionally, the processing unit is configured to generate an electronic signal for rotating the content based on a sensed orientation of the device.

[0053] Optionally, the processing unit is configured to generate electronic signals for moving the content based on the sensed orientation of the device.

[0054] Optionally, the device further includes a movement sensor for sensing movement of the device, the processing unit being configured to generate an electronic signal for varying content displayed by the image display device based on the sensed movement of the device.

[0055] Optionally, the processing unit is configured to generate an electronic signal for altering the content by moving the content based on the sensed movement of the device.

[0056] Optionally, the device is a handheld device.

[0057] Optionally, the handheld device includes a mobile phone, a smart phone, a personal digital assistant (PDA), or a tablet.

[0058] Optionally, the allocated first area of ​​the screen does not display the object while the allocated first area of ​​the screen is sensitive to a user's finger action.

[0059] Optionally, the processing unit is configured to operate the screen to display the grid of dots within the allocated first area of ​​the screen.

[0060] Optionally, the processing unit is also configured to change a characteristic of one or more of the dots in response to a user touching a portion of the allocated first area of ​​the screen in which one or more of the dots are displayed.

[0061] Optionally, the processing unit is also configured to assign a second area of ​​the screen as the first button.

[0062] Optionally, the first button is a "home" button.

[0063] Optionally, the processing unit is also configured to assign a third area of ​​the screen as a second button.

[0064] Optionally, the second button is a "toggle" button.

[0065] Optionally, the processing unit is also configured to assign a fourth area of ​​the screen as a third button.

[0066] Optionally, the third button is a "bumper" button.

[0067] Optionally, the processing unit is also configured to assign a second area of ​​the screen as a keyboard activation button, and the processing unit is configured to operate the screen to display the keyboard in response to a user touching the assigned second area of ​​the screen.

[0068] Optionally, the apparatus further includes a wireless receiver for communication with the image display device.

[0069] Optionally, the apparatus further comprises a connector for communication with an image display device via a cable.

[0070] Optionally, the apparatus further includes a non-transitory medium storing a set of instructions, the execution of which will cause the processing unit to allocate a first area of ​​the screen for sensing a user's finger action.

[0071] Optionally, changing the content includes changing the size of the content, changing the position of the content, changing the shape of the content, changing the color of the content, replacing information within the content, increasing or decreasing the amount of information within the content, or any combination of the above.

[0072] The method includes the steps of allocating a first area of ​​a screen of the device for sensing finger actions of a user of an image display device, the image display device being configured for head-wearing by the user, the device being distinct from the image display device, and generating an electronic signal to cause a change in content displayed by the image display device based on the user's finger actions sensed by the allocated first area of ​​the screen of the device.

[0073] Optionally, the screen has a touch-sensitive area, and the allocated first area is less than a total area of ​​the touch-sensitive area.

[0074] Optionally, the allocated first area has a shape that corresponds to the shape of a screen of the image display device.

[0075] Optionally, the assigned first area has dimensions based on the brand and / or model of the device.

[0076] Optionally, the allocated first area has dimensions based on characteristics of the apparatus and characteristics of the image display device.

[0077] Optionally, the method further includes ignoring input generated by the user using a portion of the touch-sensitive area that is not part of the assigned first area and that is not part of the assigned button.

[0078] Optionally, the method further comprises generating a control signal for operating the feedback component in response to the user's finger action.

[0079] Optionally, the first area has a boundary and the user's finger action includes moving the user's finger to a location that crosses, reaches, or is within a predefined distance from the boundary.

[0080] Optionally, the first area has one or more boundaries at least partially surrounding the reference location, and the control signal is for operating the feedback component in response to the user's finger reaching a predefined distance from the reference location.

[0081] Optionally, the method further includes generating different control signals for operating the feedback component to generate different respective types of feedback based on different respective spatial relationships between one or more fingers of the user relative to the first area.

[0082] Optionally, the different types of feedback include a first tactile impulse with a first amplitude and a second tactile impulse with a second amplitude different from the first amplitude.

[0083] Optionally, the different types of feedback include a first number of tactile impulses and a second number of tactile impulses different from the first number.

[0084] Optionally, the different types of feedback include a first series of tactile impulses with a first frequency and a second series of tactile impulses with a second frequency different from the first frequency.

[0085] Optionally, the different respective spatial relationships include different distances between (1) one of the user's one or more fingers, or a point between two fingers, and (2) a reference location within the assigned first area.

[0086] Optionally, the reference location includes a centre of the assigned first area.

[0087] Optionally, the different distance exceeds a threshold.

[0088] Optionally, the different respective spatial relationships include one of the one or more fingers, or a point between two fingers, arriving at different respective distances from a boundary of the assigned first area.

[0089] Optionally, the assigned first area has a first boundary and a second boundary, and the different types of feedback include at least a first type of feedback and a second type of feedback, and the feedback component is operated to generate the first type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the first boundary, and the feedback component is operated to generate the second type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the second boundary.

[0090] Optionally, the first boundary includes a left or right boundary of the allocated first area, and the second boundary includes an upper or lower boundary.

[0091] Optionally, the control signal for operating the feedback component is based on the swipe direction.

[0092] Optionally, the method further comprises receiving an input signal associated with a pinch or unpinch action performed by a user on the allocated first area of ​​the screen.

[0093] Optionally, the electronic signal is for varying a size of content displayed by the image display device in response to an input signal associated with a pinch or unpinch action.

[0094] Optionally, the method further includes obtaining an orientation of the device from an orientation sensor, and the electronic signal is for varying content displayed by the image display device based on the input signal associated with the pinch or unpinch action and the orientation of the device.

[0095] Optionally, the content is changed by contracting or expanding the content in a first plane when a pinch or unpinch action is sensed by the assigned first area while the device is in a first orientation, and the content is changed by contracting or expanding the content in a second plane when a pinch or unpinch action is sensed by the assigned first area while the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0096] Optionally, the device has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0097] Optionally, the device has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0098] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0099] Optionally, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0100] Optionally, the first plane is perpendicular to the second plane.

[0101] Optionally, the method further comprises receiving an input signal associated with a swipe action performed by a user on the allocated first area of ​​the screen.

[0102] Optionally, the electronic signal is for moving content displayed by the image display device in response to the sensed swipe action.

[0103] Optionally, the method further includes obtaining an orientation of the device from an orientation sensor, and the electronic signal is for varying content displayed by the image display device based on the input signal associated with the swipe action and the orientation of the device.

[0104] Optionally, the content is changed by moving the content in a first plane when a swipe action is sensed by the assigned first area while the device is in a first orientation, and the content is changed by moving the content in a second plane when a swipe action is sensed by the assigned first area while the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0105] Optionally, the device has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0106] Optionally, the device has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0107] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0108] Optionally, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0109] Optionally, the first plane is perpendicular to the second plane.

[0110] Optionally, the content is within a virtual three-dimensional environment and the electronic signal is for moving the content displayed by the image display device closer to or farther away from the user when a swipe action is sensed by the assigned first area while the orientation of the device is substantially parallel to a horizontal plane.

[0111] Optionally, the content is within a virtual three-dimensional environment and the electronic signal is for moving content displayed by the image display device in a vertical plane within the three-dimensional environment when a swipe action is sensed by the assigned first area while an orientation of the device is substantially perpendicular to a horizontal plane.

[0112] Optionally, the method further includes obtaining a sensor input indicative of a sensed orientation of the device, and the electronic signal is for varying content displayed by the image display device based on the sensor input indicative of the sensed orientation of the device.

[0113] Optionally, the electronic signal is for altering the content by expanding the content in one or more directions based on sensor input indicative of a sensed orientation of the device.

[0114] Optionally, the electronic signal is for altering the content by rotating the content based on a sensor input indicative of a sensed orientation of the device.

[0115] Optionally, the electronic signal is for altering the content by moving the content based on a sensor input indicative of a sensed orientation of the device.

[0116] Optionally, the method further includes obtaining a sensor input indicative of a sensed movement of the apparatus, and the electronic signal is for varying content displayed by the image display device based on the sensor input indicative of the sensed movement of the apparatus.

[0117] Optionally, the electronic signal is for altering the content by moving the content based on a sensor input indicative of a sensed movement of the device.

[0118] Optionally, the device is a handheld device.

[0119] Optionally, the handheld device includes a mobile phone, a smart phone, a personal digital assistant (PDA), or a tablet.

[0120] Optionally, the allocated first area of ​​the screen does not display the object while the allocated first area of ​​the screen is sensitive to a user's finger action.

[0121] Optionally, the method further comprises operating the screen to display the grid of dots within the allocated first area of ​​the screen.

[0122] Optionally, the method further includes changing a characteristic of one or more of the dots in response to a user touching a portion of the allocated first area of ​​the screen in which one or more of the dots are displayed.

[0123] Optionally, the method further comprises assigning a second area of ​​the screen as the first button.

[0124] Optionally, the first button is a "home" button.

[0125] Optionally, the method further comprises assigning a third area of ​​the screen as a second button.

[0126] Optionally, the second button is a "toggle" button.

[0127] Optionally, the method further comprises assigning a fourth area of ​​the screen as a third button.

[0128] Optionally, the third button is a "bumper" button.

[0129] Optionally, the method further includes assigning a second area of ​​the screen as a keyboard activation button and, in response to a user touching the assigned second area of ​​the screen, operating the screen to display the keyboard.

[0130] Optionally, the method further comprises the step of wirelessly communicating with an image display device.

[0131] Optionally, the method further comprises communicating with an image display device via a cable.

[0132] Optionally, the device includes a non-transitory medium storing instructions, the act of allocating a first area of ​​the screen to sense a user's finger action being performed based on the instructions.

[0133] Optionally, changing the content includes changing the size of the content, changing the position of the content, changing the shape of the content, changing the color of the content, replacing information within the content, increasing or decreasing the amount of information within the content, or any combination of the above.

[0134] The article of manufacture includes a non-transitory medium storing a set of instructions, the execution of which will cause performance of a method including the steps of: allocating a first area of ​​a screen of a device to sense finger actions of a user of a visual display device, the visual display device being configured for head-wearing by a user, the device being distinct from the visual display device; and generating an electronic signal to cause a change in content displayed by the visual display device based on the user's finger actions sensed by the assigned first area of ​​the screen of the device.

[0135] Optionally, the screen has a touch-sensitive area, and the allocated first area is less than a total area of ​​the touch-sensitive area.

[0136] Optionally, the allocated first area has a shape that corresponds to the shape of a screen of the image display device.

[0137] Optionally, the assigned first area has dimensions based on the brand and / or model of the device.

[0138] Optionally, the allocated first area has dimensions based on characteristics of the apparatus and characteristics of the image display device.

[0139] Optionally, the method further includes ignoring input generated by the user using a portion of the touch-sensitive area that is not part of the assigned first area and that is not part of the assigned button.

[0140] Optionally, the method further comprises generating a control signal for operating the feedback component in response to the user's finger action.

[0141] Optionally, the first area has a boundary and the user's finger action includes moving the user's finger to a location that crosses, reaches, or is within a predefined distance from the boundary.

[0142] Optionally, the first area has one or more boundaries at least partially surrounding the reference location, and the control signal is for operating the feedback component in response to the user's finger reaching a predefined distance from the reference location.

[0143] Optionally, the method further includes generating different control signals for operating the feedback component to generate different respective types of feedback based on different respective spatial relationships between one or more fingers of the user relative to the first area.

[0144] Optionally, the different types of feedback include a first tactile impulse with a first amplitude and a second tactile impulse with a second amplitude different from the first amplitude.

[0145] Optionally, the different types of feedback include a first number of tactile impulses and a second number of tactile impulses different from the first number.

[0146] Optionally, the different types of feedback include a first series of tactile impulses with a first frequency and a second series of tactile impulses with a second frequency different from the first frequency.

[0147] Optionally, the different respective spatial relationships include different distances between (1) one of the user's one or more fingers, or a point between two fingers, and (2) a reference location within the assigned first area.

[0148] Optionally, the reference location includes a centre of the assigned first area.

[0149] Optionally, the different distance exceeds a threshold.

[0150] Optionally, the different respective spatial relationships include one of the one or more fingers, or a point between two fingers, arriving at different respective distances from a boundary of the assigned first area.

[0151] Optionally, the assigned first area has a first boundary and a second boundary, and the different types of feedback include at least a first type of feedback and a second type of feedback, and the feedback component is operated to generate the first type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the first boundary, and the feedback component is operated to generate the second type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the second boundary.

[0152] Optionally, the first boundary includes a left or right boundary of the allocated first area, and the second boundary includes an upper or lower boundary.

[0153] Optionally, the control signal for operating the feedback component is based on the swipe direction.

[0154] Optionally, the method further comprises receiving an input signal associated with a pinch or unpinch action performed by a user on the allocated first area of ​​the screen.

[0155] Optionally, the electronic signal is for varying a size of content displayed by the image display device in response to an input signal associated with a pinch or unpinch action.

[0156] Optionally, the method further includes obtaining an orientation of the device from an orientation sensor, and the electronic signal is for varying content displayed by the image display device based on the input signal associated with the pinch or unpinch action and the orientation of the device.

[0157] Optionally, the content is changed by contracting or expanding the content in a first plane when a pinch or unpinch action is sensed by the assigned first area while the device is in a first orientation, and the content is changed by contracting or expanding the content in a second plane when a pinch or unpinch action is sensed by the assigned first area while the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0158] Optionally, the device has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0159] Optionally, the device has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0160] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0161] Optionally, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0162] Optionally, the first plane is perpendicular to the second plane.

[0163] Optionally, the method further comprises receiving an input signal associated with a swipe action performed by a user on the allocated first area of ​​the screen.

[0164] Optionally, the electronic signal is for moving content displayed by the image display device in response to the sensed swipe action.

[0165] Optionally, the method further includes obtaining an orientation of the device from an orientation sensor, and the electronic signal is for varying content displayed by the image display device based on the input signal associated with the swipe action and the orientation of the device.

[0166] Optionally, the content is changed by moving the content in a first plane when a swipe action is sensed by the assigned first area while the device is in a first orientation, and the content is changed by moving the content in a second plane when a swipe action is sensed by the assigned first area while the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0167] Optionally, the device has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0168] Optionally, the device has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0169] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0170] Optionally, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0171] Optionally, the first plane is perpendicular to the second plane.

[0172] Optionally, the content is within a virtual three-dimensional environment and the electronic signal is for moving the content displayed by the image display device closer to or farther away from the user when a swipe action is sensed by the assigned first area while the orientation of the device is substantially parallel to a horizontal plane.

[0173] Optionally, the content is within a virtual three-dimensional environment and the electronic signal is for moving content displayed by the image display device in a vertical plane within the three-dimensional environment when a swipe action is sensed by the assigned first area while an orientation of the device is substantially perpendicular to a horizontal plane.

[0174] Optionally, the method further includes obtaining a sensor input indicative of a sensed orientation of the device, and the electronic signal is for varying content displayed by the image display device based on the sensor input indicative of the sensed orientation of the device.

[0175] Optionally, the electronic signal is for altering the content by expanding the content in one or more directions based on sensor input indicative of a sensed orientation of the device.

[0176] Optionally, the electronic signal is for altering the content by rotating the content based on a sensor input indicative of a sensed orientation of the device.

[0177] Optionally, the electronic signal is for altering the content by moving the content based on a sensor input indicative of a sensed orientation of the device.

[0178] Optionally, the method further includes obtaining a sensor input indicative of a sensed movement of the apparatus, and the electronic signal is for varying content displayed by the image display device based on the sensor input indicative of the sensed movement of the apparatus.

[0179] Optionally, the electronic signal is for altering the content by moving the content based on a sensor input indicative of a sensed movement of the device.

[0180] Optionally, the device is a handheld device.

[0181] Optionally, the handheld device includes a mobile phone, a smart phone, a personal digital assistant (PDA), or a tablet.

[0182] Optionally, the allocated first area of ​​the screen does not display the object while the allocated first area of ​​the screen is sensitive to a user's finger action.

[0183] Optionally, the method further comprises operating the screen to display the grid of dots within the allocated first area of ​​the screen.

[0184] Optionally, the method further includes changing a characteristic of one or more of the dots in response to a user touching a portion of the allocated first area of ​​the screen in which one or more of the dots are displayed.

[0185] Optionally, the method further comprises assigning a second area of ​​the screen as the first button.

[0186] Optionally, the first button is a "home" button.

[0187] Optionally, the method further comprises assigning a third area of ​​the screen as a second button.

[0188] Optionally, the second button is a "toggle" button.

[0189] Optionally, the method further comprises assigning a fourth area of ​​the screen as a third button.

[0190] Optionally, the third button is a "bumper" button.

[0191] Optionally, the method further includes assigning a second area of ​​the screen as a keyboard activation button and, in response to a user touching the assigned second area of ​​the screen, operating the screen to display the keyboard.

[0192] Optionally, the method further comprises the step of wirelessly communicating with an image display device.

[0193] Optionally, the method further comprises communicating with an image display device via a cable.

[0194] Optionally, the article of manufacture includes instructions for allocating a first area of ​​the screen for sensing a user's finger action.

[0195] Optionally, changing the content includes changing the size of the content, changing the position of the content, changing the shape of the content, changing the color of the content, replacing information within the content, increasing or decreasing the amount of information within the content, or any combination of the above.

[0196] The computing device includes a proximity sensing display and one or more processors operably coupled to the proximity sensing display and communicatively coupled to the wearable computing system, wherein the one or more processors receive data from the wearable computing system indicating whether the proximity sensing display is visible to a user of the wearable computing system, monitor the data received from the wearable computing system for a change in user visibility of the proximity sensing display, and are configured to switch between (i) a first presentation mode, in which the one or more processors are configured to operate the proximity sensing display such that a first load is placed on a power supply of the computing device, and (ii) a second presentation mode, in which the one or more processors are configured to operate the proximity sensing display such that a second load is placed on a power supply of the computing device, the second load being less than the first load.

[0197] Optionally, the first presentation mode is one in which the one or more processors are configured to present a graphical user interface on a proximity-sensing display.

[0198] Optionally, the second presentation mode is one in which the one or more processors are configured to present a limited version of the graphical user interface on the proximity sensing display.

[0199] Optionally, the first presentation mode is one in which the one or more processors are configured to operate the proximity sensing display such that the content is presented at a first brightness level, and the second presentation mode is one in which the one or more processors are configured to operate the proximity sensing display such that the content is presented at a second brightness level that is lower than the first brightness level.

[0200] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0201] Optionally, the computing device is configured to perform one or more of the operations described in one or both of the methods described immediately below.

[0202] The computer-implemented method includes receiving a first set of data from the wearable computing system indicating whether a proximity sensing display is visible to a user of the wearable computing system at a first time point; receiving a second set of data from the wearable computing system indicating whether the proximity sensing display is visible to a user of the wearable computing system at a second, later time point; determining that a change in user visibility of the proximity sensing display has occurred based on receiving the first and second sets of data from the wearable computing system; and in response to determining that a change in user visibility of the proximity sensing display has occurred, switching between (i) a first presentation mode, in which a graphical user interface is presented on the proximity sensing display, and (ii) a second presentation mode, in which the proximity sensing display consumes less power than the first presentation mode.

[0203] Optionally, determining that a change in the user's visibility of the proximity sensing display has occurred based on receiving the first and second sets of data from the wearable computing system includes determining that the user has lost visibility of the proximity sensing display based on receiving the first and second sets of data from the wearable computing system.

[0204] Optionally, in response to determining that the user has lost visibility of the proximity sensing display, the method includes switching from the first presentation mode to a second presentation mode.

[0205] Optionally, determining that a change in the user's visibility of the proximity sensing display has occurred based on receiving the first and second sets of data from the wearable computing system includes determining that the user has restored visibility of the proximity sensing display based on receiving the first and second sets of data from the wearable computing system.

[0206] Optionally, in response to determining that the user has lost visibility of the proximity sensing display, the method includes switching from the second presentation mode to the first presentation mode.

[0207] Optionally, the method may be performed by a computing device described immediately above and / or a computing device described immediately below.

[0208] The computing device includes a proximity sensing display, a feedback component, and one or more processors operably coupled to the proximity sensing display and the feedback component and communicatively coupled to the wearable computing system, wherein the one or more processors are configured to receive data from the wearable computing system indicating whether the proximity sensing display is visible to a user of the wearable computing system, monitor the data received from the wearable computing system for changes in user visibility of the proximity sensing display, and in response to detecting a change in user visibility of the proximity sensing display, switch between (i) a first mode, in which the one or more processors are configured to present a user interface on the proximity sensing display, and (ii) a second mode, in which the one or more processors are configured to communicate one or more portions of the user interface to the user through use of the feedback component.

[0209] Optionally, the feedback component is a tactile actuator.

[0210] Optionally, the feedback component is a speaker.

[0211] Optionally, the user interface is a graphical user interface for controlling one or more functions of the wearable computing system.

[0212] Optionally, the graphical user interface includes a plurality of graphical control elements.

[0213] Optionally, the second mode is one in which the one or more processors are configured to refrain from presenting one or more of the plurality of graphical control elements on the proximity-sensing display.

[0214] Optionally, the second mode is one in which the one or more processors are configured to present a limited version of the graphical user interface on the proximity sensing display.

[0215] Optionally, the limited version of the graphical user interface on the proximity sensing display is one in which one or more of the plurality of graphical control elements are not shown, one or more of the plurality of graphical control elements have a reduced brightness level, or a combination thereof.

[0216] Optionally, the second mode is one in which the one or more processors are configured to communicate one or more locations on the proximity sensing display where one or more of the plurality of graphical control elements are presented in the first mode.

[0217] Optionally, the one or more locations on the proximity sensing display communicated in the second mode correspond to outer boundaries of one or more of the plurality of graphical control elements when presented in the first mode.

[0218] Optionally, the plurality of graphical control elements correspond to a plurality of physical user input components of a dedicated input controller associated with the wearable computing system.

[0219] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0220] Optionally, the computing device is configured to perform one or more of the operations described in the method immediately above and / or the method immediately below.

[0221] The computer-implemented method includes receiving a first set of data from the wearable computing system indicating whether a proximity sensing display is visible to a user of the wearable computing system at a first time point; receiving a second set of data from the wearable computing system indicating whether the proximity sensing display is visible to a user of the wearable computing system at a second, later time point; determining that a change in user visibility of the proximity sensing display has occurred based on receiving the first and second sets of data from the wearable computing system; and in response to determining that a change in user visibility of the proximity sensing display has occurred, switching between (i) a first mode in which a user interface is presented on the proximity sensing display and (ii) a second mode in which one or more portions of the user interface are communicated to a user of the wearable computing system through use of a feedback component.

[0222] Optionally, determining that a change in the user's visibility of the proximity sensing display has occurred based on receiving the first and second sets of data from the wearable computing system includes determining that the user has lost visibility of the proximity sensing display based on receiving the first and second sets of data from the wearable computing system.

[0223] Optionally, in response to determining that the user has lost visibility of the proximity sensing display, the method includes switching from the first mode to the second mode.

[0224] Optionally, determining that a change in the user's visibility of the proximity sensing display has occurred based on receiving the first and second sets of data from the wearable computing system includes determining that the user has restored visibility of the proximity sensing display based on receiving the first and second sets of data from the wearable computing system.

[0225] Optionally, in response to determining that the user has lost visibility of the proximity sensing display, the method includes switching from the second mode to the first mode.

[0226] Optionally, the method may be performed by one or both of the computing devices described immediately above.

[0227] The wearable computing system includes a head-mounted display configured to be worn on a user's head, one or more sensing devices configured to monitor an environment of the head-mounted display, and one or more processors operatively coupled to the head-mounted display and the one or more sensing devices and communicatively coupled to the computing device, wherein the one or more processors are configured to use data obtained from the one or more sensing devices to determine whether a display of the computing device is visible to a user of the wearable computing system, and to generate one or more messages and transmit the one or more messages to the computing device based, at least in part, on whether the display of the computing device is determined to be visible to a user of the wearable computing system.

[0228] Optionally, the one or more sensing devices include one or more cameras.

[0229] Optionally, the one or more cameras include one or more forward-facing cameras configured to capture images of an environment of a user of the wearable computing system.

[0230] Optionally, the one or more cameras include one or more inward-facing cameras configured to capture images of one or both of the user's eyes, one or more parts of the user's head or face, or a combination thereof.

[0231] Optionally, the one or more sensing devices include one or more proximity sensors.

[0232] Optionally, the wearable computing system is configured to operate in the same or similar manner as one or more of the wearable computing systems, wearable display systems, and / or image display devices described herein.

[0233] Optionally, the wearable computing system is configured to perform one or more of the operations in the methods described immediately below.

[0234] The computer-implemented method includes obtaining data from one or more sensing devices of the wearable computing system; determining whether a display of the computing device is visible to a user of the wearable computing system based on receiving the data from the one or more sensing devices of the wearable computing system; generating one or more messages based on whether the display of the computing device is determined to be visible to the user of the wearable computing system; and transmitting the one or more messages to the computing device.

[0235] Optionally, acquiring data from one or more sensing devices of the wearable computing system includes acquiring data from one or more forward-facing cameras configured to capture images of an environment of a user of the wearable computing system.

[0236] Optionally, determining whether a display of the computing device is visible to a user of the wearable computing system based on receiving data from one or more sensing devices of the wearable computing system includes determining whether a display of the computing device is shown in one or more images captured by one or more forward-facing cameras.

[0237] Optionally, acquiring data from one or more sensing devices of the wearable computing system includes acquiring data from one or more inward-facing cameras configured to capture images of one or both of the user's eyes, one or more parts of the user's head or face, or a combination thereof.

[0238] Optionally, determining whether a display of the computing device is visible to a user of the wearable computing system based on receiving data from one or more sensing devices of the wearable computing system includes determining whether the wearable computing system is worn by the user on the user's head based on one or more images captured by the one or more inward-facing cameras.

[0239] Optionally, determining whether the wearable computing system is worn by the user on the user's head based on one or more images captured by the one or more inwardly facing cameras includes determining whether the user is shown in one or more images captured by the one or more inwardly facing cameras.

[0240] Optionally, obtaining data from one or more sensing devices of the wearable computing system includes obtaining data from one or more proximity sensors.

[0241] Optionally, determining whether a display of the computing device is visible to a user of the wearable computing system based on receiving data from one or more sensing devices of the wearable computing system includes determining whether data received from one or more proximity sensors indicates that the wearable computing system is being worn by the user on the user's head.

[0242] Optionally, the method may be performed by a wearable computing system as described immediately above.

[0243] The computing device includes a proximity sensing display, a feedback component, and one or more processors operatively coupled to the proximity sensing display and the feedback component and communicatively coupled to the wearable computing system, the one or more processors receiving data indicative of a first touch input received at a first location on the proximity sensing display, receiving data indicative of a second touch input received at a second location on the proximity sensing display shortly after the first touch input, determining whether the first and second locations each fall within a particular area of ​​the proximity sensing display, and determining whether the first and second locations each fall within a particular area of ​​the proximity sensing display. The display device is configured to generate one or more messages based at least in part on one or both of the first and second touch inputs in response to determining that one or both of the locations fall within a particular area of ​​the proximity sensing display, transmit the one or more messages generated based at least in part on one or both of the first and second touch inputs to the wearable computing system, and cause the feedback component to generate feedback in response to (i) the determining that the first location falls within the particular area of ​​the proximity sensing display and (ii) the determining that the second location falls outside the particular area of ​​the proximity sensing display.

[0244] Optionally, particular regions of the proximity sensing display are associated with particular control elements.

[0245] Optionally, the particular control element is a graphical control element, and the one or more processors are further configured to display the particular control element at a location on the proximity sensing display corresponding to the particular region.

[0246] Optionally, the feedback component is a tactile actuator.

[0247] Optionally, the feedback component is a speaker.

[0248] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0249] Optionally, the computing device is configured to perform one or more of the operations in the methods described immediately below.

[0250] The computer-implemented method includes receiving data indicative of a first touch input received at a first location on the proximity sensing display; receiving data indicative of a second touch input received at a second location on the proximity sensing display shortly after the first touch input; determining whether the first and second locations each fall within a particular area of ​​the proximity sensing display; generating one or more messages based at least in part on one or both of the first and second touch inputs in response to determining that one or both of the first and second locations fall within the particular area of ​​the proximity sensing display; transmitting the one or more messages generated based at least in part on one or both of the first and second touch inputs to a wearable computing system; and providing feedback for output through a feedback component in response to (i) determining that the first location falls within the particular area of ​​the proximity sensing display and (ii) determining that the second location falls outside the particular area of ​​the proximity sensing display.

[0251] Optionally, the method may be performed by a computing device as described immediately above.

[0252] The computing device includes a proximity-sensing display, an orientation sensor configured to sense an orientation of the computing device, and one or more processors operatively coupled to the proximity-sensing display and the orientation sensor and communicatively coupled to the wearable computing system, the one or more processors configured to monitor a sequence of touch inputs received through the proximity-sensing display corresponding to any one of a plurality of different predefined gestures, and in response to detecting any one of the plurality of different predefined gestures, select a particular type of transformation associated with the detected gesture from among a plurality of different types of transformations each associated with the plurality of different predefined gestures, select a particular axis from among a plurality of different axes based on data obtained from the orientation sensor, generate a command to apply the particular type of transformation to a virtual object relative to the particular axis, and transmit the command to the wearable computing system. Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatuses described herein.

[0253] Optionally, the computing device is configured to perform one or more of the operations in the methods described immediately below.

[0254] The computer-implemented method includes obtaining data indicative of an orientation of a computing device; receiving data indicative of a sequence of touch inputs received through a proximity-sensing display of the computing device; determining that the sequence of touch inputs received through the proximity-sensing display corresponds to a particular gesture; in response to determining that the sequence of touch inputs received through the proximity-sensing display corresponds to the particular gesture, selecting a particular type of transformation associated with the particular gesture from among a plurality of different types of transformations each associated with a plurality of different predetermined gestures; selecting a particular axis from among a plurality of different axes based on an orientation of the device; and generating a command to apply the particular type of transformation to a virtual object relative to the particular axis.

[0255] Optionally, the plurality of different types of transformations includes one or more of a rotation, a translation, and a resize.

[0256] Optionally, the method may be performed by a computing device as described immediately above.

[0257] The computing device includes a proximity sensing display and one or more processors operably coupled to the proximity sensing display and communicatively coupled to the wearable display system, wherein the one or more processors are configured to present a particular piece of content on the proximity sensing display, monitor a sequence of touch inputs received through the proximity sensing display corresponding to any one of a plurality of different predetermined gestures during the presentation of the particular piece of content on the proximity sensing display, and in response to detecting any one of the plurality of different predetermined gestures during the presentation of the particular piece of content on the proximity sensing display, generate one or more messages indicating that the presentation of the particular piece of content should be handed off to the wearable display system, transmit the one or more messages to the wearable display system, and discontinue the presentation of the particular piece of content on the proximity sensing display.

[0258] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0259] Optionally, the computing device is configured to perform one or more of the operations in the methods described immediately below.

[0260] The computer-implemented method includes presenting a particular piece of content on a proximity sensing display; receiving data indicative of a sequence of touch inputs received through the proximity sensing display while presenting the particular piece of content on the proximity sensing display; determining that the sequence of touch inputs received through the proximity sensing display corresponds to a particular gesture; in response to determining that the sequence of touch inputs received through the proximity sensing display corresponds to a particular gesture, generating one or more messages indicating that presentation of the particular piece of content should be handed off to a wearable display system; transmitting the one or more messages to the wearable display system; and interrupting presentation of the particular piece of content on the proximity sensing display.

[0261] Optionally, determining that the sequence of touch inputs received through the proximity sensing display corresponds to a particular gesture includes determining that the sequence of touch inputs received through the proximity sensing display corresponds to a gesture in which a user drags one or more of their fingers from a location on the proximity sensing display associated with a particular piece of content to an edge of the proximity sensing display.

[0262] Optionally, determining that the sequence of touch inputs received through the proximity sensing display corresponds to a particular gesture includes determining that the sequence of touch inputs received through the proximity sensing display corresponds to a gesture in which a user pinches two or more of their fingers together at a particular location on the proximity sensing display associated with a particular piece of content, and subsequently moves the two or more pinched together fingers away from the particular location.

[0263] Optionally, a gesture in which a user pinches two or more of their fingers together at a particular location on a proximity sensing display associated with a particular piece of content and then moves the two or more pinched together fingers away from the particular location includes a gesture in which a user pinches two or more of their fingers together at a particular location on a proximity sensing display associated with a particular piece of content and then moves the two or more pinched together fingers to an edge of the proximity sensing display.

[0264] Optionally, a gesture in which a user pinches two or more of their fingers together at a particular location on a proximity sensing display associated with a particular piece of content and then moves the two or more pinched together fingers away from the particular location includes a gesture in which a user pinches two or more of their fingers together at a particular location on a proximity sensing display associated with a particular piece of content and then lifts the two or more pinched together fingers off the proximity sensing display.

[0265] Optionally, presenting the particular piece of content on the proximity sensing display comprises presenting a scene including the particular piece of content on the proximity sensing display.

[0266] Optionally, discontinuing the presentation of the particular piece of content on the proximity sensing display comprises moving the particular piece of content out of the scene.

[0267] Optionally, the method further includes generating identification information for the particular piece of content while presenting the particular piece of content on the proximity sensing display, and transmitting the identification information for the particular piece of content to the wearable display system.

[0268] Optionally, the method further includes generating data indicative of a current location of the particular piece of content relative to the proximity sensing display while presenting the particular piece of content on the proximity sensing display, and transmitting the data indicative of the current location of the particular piece of content to the wearable display system.

[0269] Optionally, the method may be performed by a computing device as described immediately above.

[0270] The wearable computing system includes a head-mounted display configured to be worn on a user's head, one or more cameras configured to capture images of an environment in front of the user, and one or more processors operatively coupled to the head-mounted display and the one or more cameras and communicatively coupled to a computing device, the one or more processors configured to identify a particular piece of content being presented on a display of the computing device, receive one or more messages from the computing device indicating that presentation of the particular piece of content should be handed off from the display of the computing device to the head-mounted display, and in response to the one or more messages received from the computing device, determine a location within the environment in front of the user using one or more images captured by the one or more cameras, and present the particular piece of content on the head-mounted display such that it is perceived by the user as being located at the determined location within the environment in front of the user.

[0271] Optionally, the wearable computing system is configured to operate in the same or similar manner as one or more of the wearable computing systems, wearable display systems, and / or image display devices described herein.

[0272] Optionally, the wearable computing system is configured to perform one or more of the operations in the methods described immediately below.

[0273] The computer-implemented method includes identifying a particular piece of content being presented on a display of a computing device; receiving one or more messages from the computing device indicating that presentation of the particular piece of content should be handed off from the display of the computing device to the head-mounted display; acquiring one or more images of an environment in front of a user of the head-mounted display in response to receiving the one or more messages received from the computing device; identifying a location within the environment in front of the user based on the one or more acquired images; and presenting the particular piece of content on the head-mounted display such that it is perceived by the user as being located at the determined location within the environment in front of the user.

[0274] Optionally, the display of the computing device on which a particular piece of content is presented is a two-dimensional screen.

[0275] Optionally, the step of presenting the particular piece of content on the head-mounted display such that it is perceived by the user as being located at a determined location in the environment in front of the user includes the step of presenting a three-dimensional representation of the particular piece of content on the head-mounted display such that it is perceived by the user as being located at a determined location in the environment in front of the user.

[0276] Optionally, the method further comprises receiving identification information from the computing device regarding the particular piece of content.

[0277] Optionally, identifying the particular piece of content being presented on the display of the computing device includes identifying the particular piece of content being presented on the display of the computing device based on identification information received from the computing device.

[0278] Optionally, the method further comprises receiving data from the computing device indicating a current location of the particular piece of content.

[0279] Optionally, the current location of the particular piece of content corresponds to the current location of the particular piece of content relative to a display of the computing device.

[0280] Optionally, the method may be performed by a wearable computing system as described immediately above.

[0281] The system includes a computing device configured to operate as a substitute for a dedicated input controller associated with the wearable computing system, the computing device including a proximity sensing display; and one or more processors operatively coupled to the proximity sensing display and communicatively coupled to the wearable computing system, the one or more processors configured to present a plurality of graphical control elements, respectively, at a plurality of locations on the proximity sensing display, the plurality of graphical control elements corresponding respectively to a plurality of physical user input components of the dedicated input controller, monitor a touch input received at any one of the plurality of locations on the proximity sensing display, and in response to detecting a touch input received at any one of the plurality of locations on the proximity sensing display, identify a particular one of the plurality of graphical control elements, from among the plurality of graphical control elements associated with the plurality of locations on the proximity sensing display, that is associated with the particular one of the plurality of locations on the proximity sensing display at which the detected touch input was received, generate one or more messages based on the particular graphical control element, and transmit the one or more messages to the wearable computing system.

[0282] Optionally, the physical user input components of the dedicated input controller to which the graphical control elements correspond include one or more of a "home" button, a "trigger" button, a "bumper" button, and a touchpad.

[0283] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0284] Optionally, the computing device is configured to perform one or more of the operations in the methods described immediately below.

[0285] The computer-implemented method includes providing a graphical user interface of an application launching on the computing device for output on a proximity-sensing display of the computing device; presenting through the graphical user interface a plurality of graphical control elements, each corresponding to a plurality of physical user input components of a dedicated input controller associated with the wearable computing system; receiving data indicative of touch input received at a particular location on the proximity-sensing display; determining that the particular location where the touch input was received is associated with a particular one of the plurality of graphical control elements; in response to determining that the particular location where the touch input was received is associated with the particular graphical control element, generating one or more messages based on the particular graphical control element; and transmitting the one or more messages to the wearable computing system.

[0286] Optionally, the method may be performed by a computing device as described immediately above.

[0287] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims. The present specification also provides, for example, the following items: (Item 1) 1. An apparatus for use with an image display device configured to be head-worn by a user, the apparatus comprising: Screen and a processing unit configured to allocate a first area of ​​the screen for sensing finger actions of the user; Equipped with The apparatus, wherein the processing unit is configured to generate an electronic signal to cause a change in content displayed by the image display device based on the user's finger action sensed by an assigned first area of ​​a screen of the apparatus. (Item 2) Item 1, the device of item 1, wherein the screen has a touch-sensitive area, and the allocated first area is less than a total area of ​​the touch-sensitive area. (Item 3) Item 1 , the device of item 1 , wherein the processing unit is also configured to operate a feedback component in response to the user's finger action. (Item 4) Item 4. The apparatus of item 3, wherein the first area has a boundary, and the user's finger action includes moving the user's finger to a location that crosses, reaches, or is within a predefined distance of the boundary. (Item 5) 4. The apparatus of claim 3, wherein the processing unit is configured to operate the feedback component to generate different types of feedback based on different individual spatial relationships between one or more fingers of the user relative to the first area. (Item 6) 6. The apparatus of claim 5, wherein the different individual spatial relationships include different distances between (1) one of the user's one or more fingers, or a point between two fingers, and (2) a reference location within the assigned first area. (Item 7) 6. The apparatus of claim 5, wherein the different respective spatial relationships include one of the one or more fingers, or a point between two fingers, reaching different respective distances from a boundary of the assigned first area. (Item 8) the assigned first area has a first boundary and a second boundary; the different types of feedback include at least a first type of feedback and a second type of feedback; the processing unit is configured to operate the feedback component to generate the first type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance of the first boundary; the processing unit is configured to operate the feedback component to generate the second type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance of the second boundary. Item 5. The apparatus according to item 5. (Item 9) the processing unit is configured to obtain an input signal associated with a pinch or unpinch action performed on an assigned first area of ​​the screen; the processing unit is configured to generate an electronic signal for varying a size of content displayed by the image display device based on an input signal associated with the pinch or unpinch action. Item 1. The device according to item 1. (Item 10) 10. The device of claim 9, further comprising an orientation sensor for sensing an orientation of the device, the processing unit also configured to generate an electronic signal for changing content displayed by the image display device based on an input signal associated with the pinch or unpinch action and the sensed orientation of the device. (Item 11) the processing unit is configured to generate an electronic signal to cause the content to contract or expand in a first plane when the pinch or unpinch action is sensed by the device while the device is in a first orientation; the processing unit is configured to generate an electronic signal to cause the content to contract or expand in a second plane when the pinch or unpinch action is sensed by the device while the device is in a second orientation different from the first orientation, the second plane being different from the first plane; Item 11. The apparatus according to item 10. (Item 12) the device has the first orientation when a longitudinal axis of the device forms an angle of less than 45° with a horizontal plane; the device has the second orientation when a longitudinal axis of the device forms an angle of less than 45° with a vertical plane. Item 12. The device according to item 11. (Item 13) Item 14. The device of item 1, wherein the processing unit is configured to obtain an input signal associated with a swipe action performed on an assigned first area of ​​the screen. Item 14. The device of item 13, wherein the processing unit is configured to generate an electronic signal for changing the content by moving the content in response to an input signal associated with the swipe action. (Item 15) Item 14. The device of item 13, further comprising an orientation sensor for sensing an orientation of the device, wherein the processing unit is configured to generate an electronic signal for changing content displayed by the image display device based on an input signal associated with the swipe action and the sensed orientation of the device. (Item 16) the processing unit is configured to generate an electronic signal to move the content in a first plane when the swipe action is sensed by the device while the device is in a first orientation; the processing unit is configured to generate an electronic signal to move the content in a second plane when the swipe action is sensed by the device while the device is in a second orientation different from the first orientation, the second plane being different from the first plane; Item 16. The apparatus according to item 15. (Item 17) the device has the first orientation when a longitudinal axis of the device forms an angle of less than 45° with a horizontal plane; the device has the second orientation when a longitudinal axis of the device forms an angle of less than 45° with a vertical plane. Item 17. The device according to item 16. (Item 18) Item 16. The device of item 15, wherein the content is within a virtual three-dimensional environment, and the processing unit is configured to generate an electronic signal for changing the content displayed by the image display device by moving the content closer to or farther away from the user when the swipe action is sensed by the device while an orientation of the device is substantially parallel to a horizontal plane. (Item 19) Item 16. The device of item 15, wherein the content is within a virtual three-dimensional environment, and the processing unit is configured to generate an electronic signal for moving content displayed by the image display device in a vertical plane within the three-dimensional environment when the swipe action is sensed by the device while an orientation of the device is generally perpendicular to a horizontal plane. (Item 20) 2. The device of claim 1, further comprising an orientation sensor for sensing an orientation of the device, wherein the processing unit is configured to generate the electronic signal to (1) expand the content in one or more directions, (2) rotate the content, or (3) move the content based on the sensed orientation of the device. (Item 21) 2. The device of claim 1, wherein the device is a handheld device. (Item 22) Item 1 , the device of item 1 , wherein the processing unit is also configured to assign a second area of ​​the screen as a first button. (Item 23) Item 1, wherein the processing unit is also configured to assign a second area of ​​the screen as a keyboard activation button, and the processing unit is configured to operate the screen to display a keyboard in response to the user touching the assigned second area of ​​the screen. (Item 24) 2. The device of claim 1, wherein the change in the content includes a change in size of the content, a change in position of the content, a change in shape of the content, a change in color of the content, replacing information within the content, increasing or decreasing an amount of information in the content, or any combination of the foregoing. (Item 25) 1. A method comprising: allocating a first area of ​​a screen of a device for sensing finger actions of a user of a visual display device, the visual display device being configured for head-worn by a user, the device being distinct from the visual display device; generating electronic signals to effect a change in content displayed by the image display device based on the user's finger actions sensed by an assigned first area of ​​a screen of the device; A method comprising: (Item 26) An article of manufacture comprising a non-transitory medium storing a set of instructions, the execution of which causes a method to be performed, the method comprising: allocating a first area of ​​a screen of a device for sensing finger actions of a user of a visual display device, the visual display device being configured for head-worn by a user, the device being distinct from the visual display device; generating electronic signals to effect a change in content displayed by the image display device based on the user's finger actions sensed by an assigned first area of ​​a screen of the device; Including the product. [Brief description of the drawings]

[0288] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. To better understand how to obtain the above-listed and other advantages and objects of the various embodiments of the present disclosure, briefly described above, further detailed embodiments of the present disclosure will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered limiting of its scope, the present disclosure will be explained and described with additional specificity and detail through the use of the accompanying drawings.

[0289] [Figure 1] FIG. 1 illustrates another image display system having an image display device according to some embodiments.

[0290] [Diagram 2] FIG. 2 illustrates another image display system having an image display device according to another embodiment.

[0291] [Diagram 3] FIG. 3 illustrates another image display system having an image display device according to another embodiment.

[0292] [Figure 4] FIG. 4 illustrates another image display system having an image display device according to another embodiment.

[0293] [Diagram 5] FIG. 5 illustrates an image display device that displays frames at multiple depth planes.

[0294] [Figure 6]FIG. 6 illustrates a device having a touch-sensitive screen with an area allocated to allow a user to provide input for a visual display device.

[0295] [Figure 7A] 7A-7F illustrate examples of different alternative allocated areas of the screen of the device of FIG. [Figure 7B] 7A-7F illustrate examples of different alternative allocated areas of the screen of the device of FIG. [Figure 7C] 7A-7F illustrate examples of different alternative allocated areas of the screen of the device of FIG. [Figure 7D] 7A-7F illustrate examples of different alternative allocated areas of the screen of the device of FIG. [Figure 7E] 7A-7F illustrate examples of different alternative allocated areas of the screen of the device of FIG. [Figure 7F] 7A-7F illustrate examples of different alternative allocated areas of the screen of the device of FIG.

[0296] [Figure 8A] FIG. 8A illustrates an example of visual feedback generated in response to a user touching a portion of the allocated area of ​​the screen of the device of FIG. 6 with one finger.

[0297] [Figure 8B] FIG. 8B illustrates an example of visual feedback generated in response to a user touching a portion of the allocated area of ​​the screen of the device of FIG. 6 with two fingers.

[0298] [Figure 9A] FIG. 9A illustrates an example of a keyboard that may be displayed by the device of FIG.

[0299] [Figure 9B]FIG. 9B illustrates another embodiment of a keyboard that can be displayed by the device of FIG.

[0300] [Figure 10A] FIG. 10A illustrates an example of a one finger swipe performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0301] [Figure 10B] FIG. 10B illustrates another example of a one finger swipe performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0302] [Figure 11A] FIG. 11A illustrates an example of a one finger touch movement performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0303] [Figure 11B] FIG. 11B illustrates an example of a two finger touch movement performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0304] [Figure 12A] FIG. 12A illustrates an example of a two finger pinch performed on a designated area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0305] [Figure 12B] FIG. 12B illustrates an example of a two finger unpinch performed on a designated area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0306] [Figure 13] FIG. 13 illustrates an example of radial movement of two fingers performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as finger action input for the image display device.

[0307] [Figure 14] FIG. 14 illustrates an example of a long press touch performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0308] [Figure 15] FIG. 15 illustrates an example of a tap performed on an assigned area of ​​the screen of the device of FIG. 6 and detectable by the device as a finger action input for the image display device.

[0309] [Figure 16A] FIG. 16A illustrates an example of finger actions performed on an area of ​​the screen of the device of FIG. 6 that has been assigned as a button and is detectable by the device as an input for a visual display device.

[0310] [Figure 16B] FIG. 16B illustrates an example of a combination of different finger action inputs that are detectable by the screen area of ​​the device of FIG. 6 as a combined input for an image display device.

[0311] [Figure 17] FIG. 17 illustrates the feedback feature of the device of FIG.

[0312] [Figure 18] FIG. 18 illustrates an algorithm or method for providing feedback.

[0313] [Figure 19A] FIG. 19A illustrates example finger actions and corresponding effects on displayed content performed on the device of FIG. 6 while the device is in a face-up orientation.

[0314] [Figure 19B] FIG. 19B illustrates example finger actions and corresponding effects on displayed content performed on the device of FIG. 6 while the device is in an upright orientation.

[0315] [Figure 20] FIG. 20 illustrates a method according to some embodiments.

[0316] [Figure 21] FIG. 21 illustrates an area of ​​the screen of the device of FIG. 6 downloading an application in the form of a set of instructions from a server.

[0317] [Figure 22] FIG. 22 illustrates a specialized processing system, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0318] Various embodiments of the present disclosure are directed to methods, apparatus, and articles of manufacture for providing input for a head-mounted video imaging device. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0319] Various embodiments are described below with reference to the figures. It should be noted that the figures are not drawn to scale, and elements of similar structure or function are represented by similar reference numbers throughout the figures. It should also be noted that the figures are intended only to facilitate the description of the embodiments. They are not intended as a comprehensive description of the invention or as limitations on the scope of the invention. In addition, the illustrated embodiment does not necessarily have to have all the aspects or advantages shown. An aspect or advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiment even if not so illustrated or not so explicitly described.

[0320] The following description relates to illustrative VR, AR, and / or MR systems with which the embodiments described herein may be practiced, however, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the illustrative examples disclosed herein. (Problem and solution description)

[0321] In some cases, the head mounted image display device may have a dedicated input controller to allow a user of the head mounted image display device to provide input. The dedicated input controller may be specific to a particular model of the head mounted image display device and may be unique for each brand and model of image display device. Such a dedicated input controller may allow a user of the head mounted image display device to type text by displaying a keyboard on the screen of the head mounted image display device, and the user may type text by interacting with the keyboard as displayed on the screen of the head mounted image display device using the dedicated input controller. Such techniques of typing text may be cumbersome and may not be ideal. Also, the dedicated input controller may have limited voice and gesture control. In addition, if the dedicated input controller for a particular head mounted image display device is lost, if the dedicated input controller is damaged, or if the user forgets to bring the dedicated input controller, there is no backup controller for the user to use. Furthermore, the dedicated input controller for the head mounted image display device may not be able to transfer digital images, photos, and other media content between the user's media storage device and the head mounted image display device. The dedicated input controller may also not be able to access application stores and content, and may not be able to receive notifications from the head-mounted image display device or from the network while the head-mounted image display device is not being worn by the user.

[0322] According to one or more embodiments described herein, a device having a touch screen is provided as an input device to allow a user of a head-mounted image display device, sometimes also referred to herein as a "wearable computing system" or "wearable display system", to type input for the image display device. The device, also referred to herein more simply as a "computing device", may serve as a backup input device such that if a dedicated input controller for the head-mounted image display device is unavailable, the device may be used by the user to type input instead. The device may alternatively also be used as the main or sole input device for the image display device. The device may be used as the main device for inputting text, or alternatively as a replacement for a dedicated input controller for inputting text. Typing text through the touch screen of the device while the user views the screen of the device through a transparent portion of the head-mounted image display device is more convenient and efficient than displaying a keyboard on the viewing screen of the image display device while the user types text using a dedicated input controller. Also, in embodiments in which the apparatus is implemented using a user's mobile phone, the apparatus may also facilitate the transfer of digital images and photographs and other media content between the user's phone and the image display device. In addition, in some embodiments, the apparatus described herein allows a user of a head-mounted image display device to access application stores and content. Furthermore, in some embodiments, the apparatus described herein may receive notifications from the head-mounted image display device or from a network when the head-mounted image display device is not being worn by the user.

[0323] 1-4 illustrate various components of an image display system 100 in various embodiments. The image display system 100 includes an image display device 101 and an apparatus 200 for providing input for the image display device 101. The apparatus 200 will be described in more detail below. The image display device 101 may be a VR device, an AR device, an MR device, or any other type of display device. The image display device 101 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102 such that the display subsystem 110 is positioned in front of the eye of the end user 50, and a speaker 106 carried by the frame structure 102 such that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to the other ear canal of the end user 50 to provide stereo / adjustable sound control). The display subsystem 110 is designed to present to the eyes of the end user 50 light patterns that can be comfortably perceived as an augmentation to physical reality, with a high level of image quality and three-dimensional perception, and is also capable of presenting two-dimensional content. The display subsystem 110 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene.

[0324] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display, through which the user can view light from real objects directly and through a transparent (or semi-transparent) element. The transparent element, often referred to as a "combiner," superimposes the light from the display over the user's view of the real world. To this end, the display subsystem 110 comprises a partially transparent display. The display is positioned within the field of view of the end user 50, between the eyes of the end user 50 and the surrounding environment, such that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.

[0325] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, whereby light is combined with direct light from the surrounding environment and transmitted from the display to the user's eye 50. The projection subsystem may be a fiber optic scanning-based projection device, and the display may be a waveguide-based display into which the light scanned from the projection subsystem is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. These layers in the light field may be stacked close enough together to appear persistent to the human accessory visual system (i.e., one layer is within the noise signal cone of an adjacent layer). Additionally or alternatively, photographic elements may be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field, even when those layers are more sparsely stacked (i.e., one layer is outside the noise signal cone of an adjacent layer). The display subsystem 110 may be monocular or binocular.

[0326] The image display device 101 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the eye position and interocular distance of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes), or any combination of the foregoing. Many of these sensors operate under the assumption that the frame 102, onto which they are affixed, is substantially fixed to the user's head, eyes, and ears, in turn.

[0327] The image display device 101 may also include a user orientation detection module. The user orientation module may detect the instantaneous position of the end user's 50 head 54 (e.g., via a sensor coupled to the frame 102) and predict the position of the end user's 50 head 54 based on the position data received from the sensor. Detecting the instantaneous position of the end user's 50 head 54 facilitates determining the specific real object the end user 50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that real object, and further providing an indication of the location where the virtual object will be displayed. The user orientation module may also track the eyes of the end user 50 based on tracking data received from the sensor.

[0328] The image display device 101 may also include a control subsystem, which may take any of a wide variety of forms. The control subsystem may include a number of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphic processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), e.g., field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0329] The control subsystem of image display device 101 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a 3D database for storing 3D scene data. The CPU may control the overall operation, while the GPU may render frames from the 3D data stored in the 3D database (i.e., convert the 3D scene into a 2D image) and store these frames in the frame buffer. One or more additional integrated circuits may control the loading of frames into and / or from the frame buffer and the operation of the image projection assembly of display subsystem 110.

[0330] The various processing components of the image display device 101 may be physically contained within a distributed subsystem. For example, as illustrated in Figures 1-4, the image display device 101 may include a local processing and data module 130 that is operatively coupled to the display subsystem 110 and sensors, such as by wired leads or wireless connectivity 136. The local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (Figure 1), fixedly attached to a helmet or hat 56 (Figure 2), removably attached to a torso 58 of the end user 50 (Figure 3), or removably attached to a waist 60 of the end user 50 in a belt-type configuration (Figure 4). The image display device 101 may also include a remote processing module 132 and a remote data repository 134 operably coupled to the local processing and data module 130, such as by wired conductors or wireless connectivity 138, 140, such that these remote modules 132, 134 are operably coupled to each other and available as resources to the local processing and data module 130.

[0331] The local processing and data module 130 may comprise a power efficient processor or controller and digital memory such as flash memory, both of which may be utilized to aid in the processing, caching, and storage of data captured from the sensors and / or obtained and / or processed using the remote processing module 132 and / or the remote data repository 134, possibly for passing to the display subsystem 110 after processing or retrieval. The remote processing module 132 may comprise one or more relatively powerful processors or controllers configured to analyze and process the data and / or image information. The remote data repository 134 may comprise a relatively large digital data storage facility that may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130, allowing for fully autonomous use from any remote module.

[0332] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports to provide wire or optical communication, or one or more wireless interfaces or ports, such as via RF, microwave, and IR, to provide wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. In still further implementations, the wired and wireless communication options may differ from those illustrated in FIGS. 1-4. Thus, the particular options of wired or wireless communication should not be considered limiting.

[0333] In some embodiments, the user orientation module is contained within the local processing and data module 130, while the CPU and GPU are contained within the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained within the local processing and data module 130. The 3D database can be associated with a remote data repository 134 or located locally.

[0334] Some image display systems (e.g., VR systems, AR systems, MR systems, etc.) use multiple volume phase holograms, surface relief holograms, or light directing optical elements that are embedded with depth plane information and generate images that appear to originate from individual depth planes. In other words, a diffraction pattern or diffractive optical element ("DOE") may be embedded within or imprinted / embossed on a light directing optical element ("LOE"; e.g., a planar waveguide) such that collimated light (a light beam with a substantially planar wavefront) intersects with the diffraction pattern at multiple locations as it is substantially totally internally reflected along the LOE and exits toward the user's eye. The DOE is configured such that the light exiting the LOE through it is converged such that it appears to originate from a particular depth plane. The collimated light may be generated using an optical focusing lens ("concentrator").

[0335] For example, a first LOE may be configured to deliver collimated light to the eye that appears to originate from an optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopters). It should be understood that by using stacked LOE assemblies, multiple depth planes may be created, with each LOE configured to display an image that appears to originate from a particular depth plane. It should be understood that a stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Furthermore, N, 2N, or 3N stacked LOEs may be used to generate RGB color images at N depth planes.

[0336] To present 3-D virtual content to a user, the image display system 100 (e.g., a VR system, an AR system, an MR system, etc.) projects images of the virtual content into the user's eye so that they appear to originate from different depth planes in the Z direction (i.e., orthogonally away from the user's eye). In other words, the virtual content may not only vary in the X and Y directions (i.e., 2D planes, orthogonal to the central visual axis of the user's eye), but may also appear to vary in the Z direction so that the user may perceive objects as being very close or at infinite distance or any distance in between. In other embodiments, the user may perceive multiple objects at different depth planes simultaneously. For example, the user may see a virtual dragon appearing from infinity and running towards the user. Alternatively, the user may see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at an arm's length (approximately 1 meter) from the user at the same time.

[0337] The multi-plane focusing system creates the perception of variable depth by projecting images onto some or all of a number of depth planes located at separate fixed distances in the Z direction from the user's eyes. Now referring to FIG. 5, it should be understood that the multi-plane focusing system may display frames at fixed depth planes 150 (e.g., the six depth planes 150 shown in FIG. 5). Although an MR system can include any number of depth planes 150, one exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. In generating virtual content in one or more of the six depth planes 150, a 3-D perception is created such that the user perceives one or more virtual objects at variable distances from the user's eyes. Given that the human eye is more sensitive to objects that are closer than objects that appear to be farther away, more depth planes 150 are generated closer to the eyes, as shown in FIG. 5. In other embodiments, the depth planes 150 may be placed at equal distances away from each other.

[0338] Depth plane positions 150 may be measured in diopters, a unit of refractive power equal to the inverse of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopters away, depth plane 2 may be 0.3 diopters away, depth plane 3 may be 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate depth planes 150 at other distances / diopters. Thus, generating virtual content at strategically placed depth planes 150 allows a user to perceive virtual objects in three dimensions. For example, a user may perceive a first virtual object as being closer to him when it is displayed in depth plane 1 while another virtual object appears at infinity in depth plane 6. Alternatively, the virtual object may be displayed first in depth plane 6, then in depth plane 5, etc., until the virtual object appears very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be centered on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup 0.5 meters away from the user, all six depth planes may be generated at various cross sections of the coffee cup, giving the user a highly granular 3-D view of the coffee cup.

[0339] In some embodiments, image display system 100 (e.g., a VR system, an AR system, an MR system, etc.) may function as a multi-plane focal system. In other words, all six LOEs may be illuminated simultaneously such that images appearing to arise from six fixed depth planes are generated in rapid succession, with the light source rapidly transmitting image information to LOE1, then LOE2, then LOE3, etc. For example, a portion of a desired image, including an image of the sky at optical infinity, may be injected at time 1, and an LOE that retains collimation of the light (e.g., depth plane 6 from FIG. 5) may be utilized. An image of a closer tree branch may then be injected at time 2, and an LOE configured to create an image that appears to arise from a depth plane 10 meters away (e.g., depth plane 5 from FIG. 5) may be utilized. An image of a pen may then be injected at time 3, and an LOE configured to create an image that appears to arise from a depth plane 1 meter away may be utilized. This type of paradigm can be repeated in a fast time-sequential (eg, 360 Hz) manner so that the user's eyes and brain (eg, visual cortex) perceive the inputs as all being part of the same image.

[0340] The image display system 100 may project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth plane) and generate images for a 3-D experience / scenario. As used herein, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) emitted from a light source. Generating images that appear to originate from various depth planes conforms to the vergence and accommodation of the user's eyes with respect to the images and minimizes or eliminates vergence-accommodation conflicts.

[0341] As described and shown in FIGS. 1-4, the image display system 100 includes an apparatus 200 for providing input for the image display device 101. The apparatus 200 allows a user 50 of the image display device 101 to type in user input while the user 50 is viewing content displayed by the image display device 101. In one implementation, the apparatus 200 may be a handheld device (mobile phone (e.g., smartphone), tablet, IPAD, minipad, etc.) configured to interact with the image display device 101. In the example shown in FIGS. 1-4, the apparatus 200 interacts with the image display device 101 by communicating with the processing module 130 of the image display device 101. The communication between the apparatus 200 and the processing module 130 of the image display device 101 may be achieved via a wireless connection or a wired connection. In other embodiments, the apparatus 200 may also communicate with a remote processing module 132 and / or a remote data repository 134 via a wireless connection.

[0342] FIG. 6 illustrates an example of an apparatus 200. The apparatus 200 is for use with an image display device 101 configured to be head-worn by a user 50. In particular, the apparatus 200 is configured as an input device / controller for enabling the user 50 to provide user input using his / her finger while viewing an image displayed by the display device 101. In some cases, the apparatus 200 may be considered as part of the image display system 100. The apparatus includes a screen 202 and a processing unit 204. The processing unit 204 is configured to allocate a first area 206 of the screen 202 for sensing finger actions of the user 50. The processing unit 204 is configured to detect different finger actions performed on the allocated first area 206, thereby enabling the allocated first area 206 to function like a touchpad. The processing unit 204 is also configured to generate an electronic signal to cause a change in content displayed by the image display device 101 based on a finger action of the user 50 sensed by the assigned first area of ​​the screen 202 of the apparatus 200. The electronic signal may represent an identification of the detected finger action and / or may represent a command generated by the processing unit 204 based on the detected finger action.

[0343] In the illustrated embodiment, device 200 is a handheld device. By way of non-limiting examples, the handheld device may be a mobile phone, a smart phone, a personal digital assistant (PDA), or a tablet.

[0344] 6, the screen 202 of the device 200 has a touch-sensitive area 210, and the allocated first area 206 is less than the total area of ​​the touch-sensitive area 210. Thus, the processing unit 204 is configured to ignore input generated by the user 50 using a portion of the touch-sensitive area 210 that is not part of the allocated first area 206.

[0345] The size and shape of the first area 206 may be predefined or selectively configured by the user 50. In other embodiments, the first area 206 may have other shapes instead of the square shape shown in FIG. 6. FIGS. 7A-7F illustrate examples of different first areas 206 that may be assigned by the processing unit 204. FIG. 7A illustrates a first area 206 having a rectangular shape, with the long side of the first area 206 parallel to the short side of the device 200 and the short side of the first area 206 parallel to the long side of the device 200. In other embodiments, the first area 206 may have a rectangular shape, with the long side of the first area 206 parallel to the long side of the device 200 and the short side of the first area 206 parallel to the short side of the device 200, such as the one shown in FIG. 7B. In further embodiments, the size of the first area 206 may be smaller (or larger) than the illustrated example. For example, in other embodiments, the first area 206 may have a small size, occupying only a discrete area of ​​the screen (FIG. 7C). In further embodiments, the first area 206 may have a circular shape (FIG. 7D), a hexagonal shape (FIG. 7E), or any other polygonal shape.

[0346] In some embodiments, the first area 206 may have a shape that corresponds to an area of ​​the display screen of the image display device 101. For example, if the screen of the image display device 101 has an aspect ratio of 4:3, the first area 206 allocated by the processing unit 204 may also have an aspect ratio of 4:3 (FIG. 7F). In one implementation, the processing unit 204 of the apparatus 200 is configured to determine a shape of the display screen area of ​​the image display device 101 and determine a shape of the first area 206 based on the shape of the display screen area of ​​the image display device 101. For example, the processing unit 204 may determine that the shape of the first area 206 is the same as the shape of the display screen area of ​​the image display device 101.

[0347] In some embodiments, the processing unit 204 may be configured to determine the size of the touch-sensitive region 210 and determine the size and / or shape for which the first area 206 is assigned to sense finger actions of the user 50. For example, the processing unit 204 may determine that the entire touch-sensitive region 210 of the device 200 is 2.5 inches wide and 5.5 inches long. In such a case, the processing unit 204 may then determine that the width of the first area 206 is a constant k1 (e.g., 0.9) times the width of the touch-sensitive region 210 (which in the above example would be 0.9 x 2.5 = 2.25 inches). Similarly, the processing unit 204 may determine that the length of the first area 206 is a constant k2 (e.g., 0.5) times the length of the touch-sensitive region 210 (which in the above example would be 0.5 x 5.5 = 2.75 inches).

[0348] In other embodiments, the processing unit 204 may determine the brand and model of the device 200 and determine the size and / or shape for which the first area 206 is assigned for sensing finger actions of the user 50. For example, a brand and model of the device 200 having a relatively larger screen size may be assigned a relatively larger first area 206 as compared to another brand and model of the device 200 having a relatively smaller screen size.

[0349] After the first area 206 is assigned to sense the finger action of the user 50, the user 50 may then use one or more fingers to type input for the image display system 100 using the assigned first area 206 of the device 200. For example, the user 50 may move a finger within the assigned first area 206 to move an object displayed by the image display device 101. The object may be any object such as a cursor, text, an image, a photo, a window, a frame, an application page, etc. As another example, the user 50 may move two fingers in the same swipe direction within the assigned first area 206 to move an object displayed by the image display device 101. As a further example, the user 50 may perform a pinch or unpinch action within the assigned first area 206 to change the size of the object (e.g., pinch to reduce the size, unpinch to increase the size). As a further example, user 50 may perform tap actions (e.g., single tap, double tap, etc.) within assigned first area 206 to select an object, deselect an object, grasp an object, ungrasp an object, etc. Thus, apparatus 200 is configured as an input device / controller to enable user 50 to provide input to interact with content displayed by image display device 101.

[0350] In the first mode of operation, the device 200 may not provide a visual display on the screen 202 while the user 50 uses the device 200 to generate input for the image display system 100. The allocated first area 206 of the screen does not display objects while the allocated first area 206 of the screen 202 senses the finger actions of the user 50. In such a case, the device 200 is configured like a touchpad. However, unlike a touchpad, the device 200 itself may be a mobile or handheld device. In some implementations, the presentation of content on the screen 202 by the device 200 may be at least partially degraded or disabled when in the first mode of operation. More specifically, when in the first mode of operation, the device 200 may, in such implementations, provide less content for visual display on the screen 202, provide reduced brightness and / or contrast content for visual display on the screen 202, reduce the brightness level of the screen 202, or take one or more other actions to reduce display functionality and conserve power. In some examples, when in the first mode of operation, device 200 may communicate the boundaries of the touchpad and / or other portions of the user interface to which the touchpad belongs through the use of one or more feedback components (e.g., haptic feedback components, speakers, etc.). In these examples, when in the first mode of operation, device 200 may communicate one or more portions of such user interface using one or more feedback components instead of or in addition to visually displaying the user interface using screen 202. In some embodiments, a representation of such a user interface or a portion thereof may be displayed by visual display device 101 while device 200 is operating in the first mode. In at least some of these embodiments, the relative location on screen 202 to which touch input is being provided may also be visually represented for the user through visual display device 101.In this manner, a user may be able to effectively interact with screen 202 while looking anywhere. In some of these embodiments, when in the first mode of operation, such a user interface, or a representation of a portion thereof, may be displayed using image display device 101 instead of, or in addition to, visually displaying the user interface, or a limited version thereof (e.g., an at least partially degraded or disabled version of the user interface) using screen 202 of device 200.

[0351] In the second mode of operation, the processing unit 204 may be configured to operate the screen 202 to display content while allowing the screen 202 to receive finger action input from the user 50. In some implementations, the quantity, brightness, and / or contrast of content provided on the screen 202 for visual display by the device 200 when in the second mode of operation may exceed the quantity, brightness, and / or contrast of content provided on the screen 202 for visual display by the device 200 when in the first mode of operation. Similarly, in some examples, the device 200 may operate the screen 202 in the second mode of operation at a higher brightness level than in the first mode of operation. For example, the processing unit 204 may operate the screen 202 to display a grid of dots 220 within the assigned first area 206 of the screen 202, such as that shown in FIG. 6. The grid of dots 220 allows the user 50 to see where the assigned first area 206 is located so that the user 50 may more accurately position his / her finger. In some cases, the processing unit 204 may also be configured to change characteristics of one or more of the dots 220 in response to the user 50 touching a portion of the assigned first area 206 of the screen 202 in which one or more of the dots 206 are displayed (FIGS. 8A-8B). In particular, FIG. 8A illustrates an example of visual feedback 230 generated by the processing unit 204 in response to the user 50 touching a portion of the assigned area 206 of the screen 202 of the device 200 with one finger. FIG. 8B illustrates an example of visual feedback 230 generated by the processing unit 204 in response to the user 50 touching a portion of the assigned area 206 of the screen 202 of the device 200 with two fingers. This allows the user 50 to confirm that the input generated by the finger action was successfully received by the processing unit 204. In the above example, the visual feedback 230 includes a circular shape and a change in the spacing of the dots 220 proximate to the touched location.In other embodiments, visual feedback 230 may include only a circular shape, without any change in spacing of dots 220. In a further embodiment, instead of a circular shape, visual feedback 230 may include an object (of any shape) generated by processing unit 204 that is positioned at the location of screen 202 touched by user 50. In still further embodiments, visual feedback 230 may not include any object overlaid with dots 220. In such a case, visual feedback 230 may be a change in color of the dots and / or a change in shape and / or size of the dots at the location where user 50 touched screen 202.

[0352] In some embodiments, the horizontal spacing of dots 220 may be the same as the vertical spacing of dots 220. In other embodiments, the horizontal spacing of dots 220 may differ from the vertical spacing of dots 220. For example, the horizontal spacing of dots 202 may be greater than or less than the vertical spacing of dots 220.

[0353] It should be noted that in the second mode, the screen of the device 200 is not limited to displaying the dots 220, and in other embodiments may display other content. For example, in other embodiments, when in the second mode, the screen of the device 200 may display several objects (e.g., photos, videos, icons, etc.) for the user 50 to select using the device 200, text boxes for the user 50 to type text, a drawing "pad" to allow the user 50 to draw things, tabs for the user 50 to select, application graphics to allow the user 50 to interact with, etc. In some implementations, a polygonal shape may be displayed within the allocated first area 206 of the screen 202. For example, in these implementations, the device may display a solid polygonal shape, an outline of a polygonal shape, or both, within the allocated first area 206 of the screen 202. In some examples, the geometry of such a polygonal shape may correspond to the geometry of the assigned first area 206 such that one or more boundaries of the polygonal shape may coincide with one or more boundaries of the assigned first area 206, respectively.

[0354] In some embodiments, the device 200 may switch between the first mode and the second mode. For example, the device 200 may switch between the first mode and the second mode based on a command received from the image display device 101. In some implementations, the image display device 101 may analyze data output from one or more of its sensing devices and determine whether the device 200 is within the field of view of the user. In some examples, such one or more sensing devices of the image display device 101 may correspond to one or more forward-facing cameras. For example, the image display device 101 may analyze images captured by its cameras and determine whether the device 200 is shown in any of the images (and thus located within the field of view of the user). If the device 200 is not detected by any of the cameras, in response, the image display device 101 may then instruct the device 200 to operate in the first mode. On the other hand, if the device 200 is detected by the camera of the image display device 101, in response thereto, the image display device 101 may then instruct the device 200 to operate in the second mode. This feature is advantageous because it may provide a power saving advantage when the user 50 is not looking at the screen of the device 200. Thus, when the user 50 is not looking at the device 200 through the screen of the image display device 101, the screen of the device 200 may operate in the first mode (displaying no content or a low level set of content), and when the user 50 is looking at the device 200 through the screen of the image display device 101, the screen of the device 200 may operate in the second mode (displaying content in its entirety). In some implementations, the image display device 101 may utilize one or more of its sensing devices other than one or more of its forward facing cameras to determine whether the device 200 is within the user's field of view.For example, the image display device 101 may determine whether the apparatus 200 is within the user's field of view based on data output from one or more proximity sensors, an inward-facing eye-tracking camera, or other sensing device of the image display device 101 capable of outputting data indicative of whether the user is wearing the image display device 101. This may be beneficial, particularly in implementations where the image display device 101 is a virtual reality (VR) headset or other wearable computing device that physically occludes a portion of the user's view of the environment when worn by the user, for the image display device 101 to instruct the apparatus 200 to switch between a first and a second operating mode based on data output from one or more such sensing devices.

[0355] Also, in some embodiments, each unit length Lh of horizontal movement input implemented in the first area 206 may cause a corresponding amount of horizontal movement Mh of the object displayed by the image display device 101. Similarly, each unit length Lv of vertical movement input implemented in the first area 206 may cause a corresponding amount of vertical movement Mv of the object displayed by the image display device 101. The horizontal movement efficiency may be represented by the ratio Mh / Lh, and the vertical movement efficiency may be represented by the ratio Mv / Lv. When the ratio Mh / Lh is 1.0, this means that one unit of horizontal movement input in the allocated first area 206 will cause the same unit of horizontal movement for the object displayed by the image display device 101. When the ratio Mh / Lh is greater than 1.0, this means that one unit of horizontal movement input in the allocated first area 206 will cause more than one unit of horizontal movement for the object displayed by the image display device 101. If the ratio Mh / Lh is less than 1.0, this means that a horizontal movement input of one unit in the allocated first area 206 will cause a horizontal movement of less than one unit for the object displayed by the image display device 101. A similar concept applies in the vertical direction. In some embodiments, the screen of the device 200 may be smaller in size compared to the screen of the image display device 101. In such cases, it may be desirable to configure the allocated first area 206 such that Mh / Lh>1 and Mv / Lv>1. This allows the user 50 to operate the relatively smaller first area 206 on the device 200 and cover a larger screen area in the image display device 101. In other embodiments, the screen of the device 200 may be larger in size compared to the screen of the image display device 101. In such cases, it may be desirable to configure the allocated first area 206 such that Mh / Lh<1 and Mv / Lv<1.This allows user 50 to operate a relatively larger first area 206 on device 200 to cover a smaller screen area in image display device 101. In other embodiments, regardless of the relative screen sizes of device 200 and image display device 101, the movement efficiency (Mh / Lh and Mv / Lv) of the allocated first area 206 may be arbitrarily set to any value.

[0356] In some embodiments, the ratio Mh / Lh may be equal to the ratio Mv / Lv. This means that the horizontal movement efficiency is the same as the vertical movement efficiency for inputs typed using the assigned first area 206. In other embodiments, the ratio Mh / Lh may be higher than the ratio Mv / Lv. This means that the horizontal movement efficiency is higher than the vertical movement efficiency for inputs typed using the assigned first area 206. This allows the user 50 to cover more horizontal space in the displayed screen of the image display device 101 by moving through less horizontal space in the assigned first area 206 (compared to the vertical space). In further embodiments, the ratio Mh / Lh may be less than the ratio Mv / Lv. This means that the horizontal movement efficiency is lower than the vertical movement efficiency for inputs typed using the assigned first area 206. This allows the user 50 to cover more vertical space within the displayed screen of the image display device 101 by moving through less vertical space within the allocated first area 206 (compared to the horizontal space).

[0357] In some embodiments, Mh / Lh may be based on the width of the screen of the image display device 101, and Mv / Lv may be based on the height of the screen of the image display device 101. Thus, Mh / Lh and Mv / Lv may be based on the aspect ratio or shape of the screen of the image display device 101. For example, if the screen of the image display device 101 has an aspect ratio (width to height ratio) of 4:3, Mh / Lh may be 1.33 times Mv / Lv. When these movement efficiencies are applied with respect to the allocated first area 206 having, for example, a square shape, a unit of horizontal movement input by the user will move an object displayed within the image display device 101 1.33 times more compared to that with respect to a unit of vertical movement. Thus, even if the assigned first area 206 (which in an embodiment has a square shape) may not correspond to the shape of the screen of the image display device 101 (which in an embodiment has an aspect ratio of 4:3), the movement efficiency of the assigned first area 206 may be selected or determined such that the user 50 may use the assigned first area 206 to cover all of the space horizontally and vertically.

[0358] In further embodiments, the horizontal movement efficiency and / or vertical movement efficiency for the assigned first area 206 may be variable. In particular, the horizontal movement efficiency and / or vertical movement efficiency for the assigned first area 206 may be a function of the input location within the assigned first area 206 and / or a function of the cursor / pointer location within the screen of the image display device 101. For example, the horizontal movement efficiency may have a first value when the input generated in the first area 206 corresponds to the center of the field of view or the center of the first area 206, and may have a second value (higher than the first value) when the input generated in the first area 206 corresponds to a location on the periphery of the field of view or the periphery of the first area 206. This feature may be desirable because it allows the user 50 to scroll or move through more content as their finger approaches the first area 206 or the left / right edge of the field of view. The scrolling speed may appear to be faster because the cursor is "moving more" through the content. In other embodiments, the second value may be lower than the first value.

[0359] In one or more embodiments described herein, the horizontal movement efficiency and / or vertical movement efficiency for the first area 206 may be predetermined, may be determined by the processing unit 204, and / or may be configurable by a user 50 of the device 200.

[0360] 6, in some embodiments, the processing unit 204 may also be configured to optionally assign the second area 240 of the screen 202 as a first button 242. In the illustrated example, the first button 242 is a "home" button. In other embodiments, the first button 242 may be other types of buttons for performing other types of functions.

[0361] The processing unit 204 may also be configured to optionally assign a third area 250 of the screen as a second button 252. In the illustrated example, the second button 252 is a "toggle" button. A toggle button may allow a user to perform a selection function and / or change a function of a cursor or pointer. In other embodiments, the second button 252 may be other types of buttons for performing other types of functions.

[0362] In addition, processing unit 204 may be configured to optionally assign a fourth area 260 of the screen as a third button 262. In the illustrated example, third button 262 is a "bumper" button. A bumper button is a multi-function button depending on the software / operating system of device 200. In some cases, a bumper button may enable user 50 to control volume, turn pages, change magnification, etc. In other embodiments, third button 262 may be other types of buttons for performing other types of functions.

[0363] In other embodiments, the processing unit 204 may allocate more than three areas of the screen as separate input areas (eg, pad areas for receiving finger movement input, buttons, controls, etc.).

[0364] Further, in some embodiments, any control of the apparatus 200 may be operable to generate input for the image display device 101. For example, in some embodiments, the volume control of the apparatus 200 may be utilized to control a speaker of the image display device 101. In one implementation, the processing unit 204 is configured to detect actuation of the volume control in the apparatus 200. In response to the detected actuation, the processing unit 204 then provides a corresponding control signal (e.g., a volume up signal, a volume down signal, a mute, etc.) for transmission to the image display device 101. The transmission of the control signal may be accomplished wirelessly or through a cable. The image display device 101 then operates its speaker according to the control signal provided by the apparatus 200.

[0365] Also, as shown in FIG. 6, in some embodiments, the processing unit 204 may provide an “image capture” button 274 to enable the user 50 to capture an image presented by the screen of the image display device 101. The image may include content displayed by the screen of the image display device 101 and / or content in the surrounding environment as viewed through the screen of the image display device 101. In one implementation, the processing unit 204 is configured to detect a press of the “image capture” button 274 and, in response, generate a control signal (for transmission to the image display device 101) to cause the image display device 101 to perform a screenshot function and capture a screenshot of the content displayed by the image display device 101. Alternatively, or in addition, the processing unit 204 may generate a control signal to cause the image display device 101 to operate its camera and take a picture of the surroundings as viewed by the camera. The image of the displayed content and the image of the surroundings may be combined by the processing unit 204 and / or the processing unit 130 to form a composite image. In some cases, the "capture image" button 274 may also be pressed to capture an image of the content displayed by the device 200. The image displayed by the device 200 may be an artificially generated graphic and / or a camera image captured by the camera of the device 200. For example, the processing unit 204 may detect a press of the "capture image" button 274 and, in response, perform a screenshot function to capture a screenshot of the content displayed by the device 200. In other embodiments, the "capture image" button 274 may also be used to capture a video. For example, the "capture image" button 274 may be pressed and held for a duration. The processing unit 204 detects that there is a press and hold of the "capture image" button 274 and activates the camera of the device 200 to start recording the video. The user 50 may again press and hold the "capture image" button 274 to stop recording the video.Thus, as soon as the processing unit 204 detects that there is a second long press on the "capture image" button 274, the processing unit 204 then operates the camera and stops recording the video. It should be noted that other techniques for generating video using the device 200 may be employed in other embodiments, and the "capture image" button 274 is merely an example of a control that may be used to generate video.

[0366] In one or more embodiments, an image or video captured by the device 200 may be transmitted by the device 200 to a network (e.g., a cloud network) for storage. In such a case, the image display device 101 may retrieve the stored image or video from the network (e.g., a cloud network) for display on its screen. Alternatively, an image or video captured by the device 200 may be transmitted to the image display device 101 by a short-range network (e.g., a Bluetooth network, Wi-Fi, etc.). Also, in some embodiments, the device 200 may be configured to retrieve media content (e.g., photos, pictures, videos, etc.) from the network. For example, another user may upload media content to the cloud network, and the user 50 may retrieve such media content by accessing the cloud network via an account associated with the device 200 and / or the image display device 101.

[0367] In some embodiments, when the user 50 is using the assigned first area 206 (simulated touchpad area) of the screen 202, the buttons 242, 252, 262 are disabled. This may have the advantage of preventing the user 50 from inadvertently touching and activating any of these buttons. In other embodiments, when the user 50 is using the assigned first area 206 of the screen 202, one or more of the buttons 242, 252, 262 are not disabled, thereby allowing the user 50 to perform simultaneous finger actions using both the first assigned area 206 (simulated touchpad area) and any of the buttons 242, 252, 262.

[0368] 6, the processing unit 204 may also be configured to optionally assign another area 270 of the screen 202 as a keyboard activation button 272, and the processing unit 204 is configured to operate the screen 202 to display a keyboard 274 in response to the user 50 touching the assigned second area 270 of the screen 202 where the keyboard activation button 272 is located (FIG. 9A). FIG. 9B illustrates another example of a keyboard 274 that may be displayed by the screen 202. The displayed keyboard 274 allows the user 50 to type text, numbers, symbols, expressions, messages, etc.

[0369] Returning to FIG. 6 , the processing unit 204 is configured to perform gesture recognition and identify finger actions performed on any or a combination of the assigned areas 206, 240, 250, 260. In one implementation, the device 200 includes a buffer, and as the user 50 types input using finger actions performed on the assigned areas, the device 200 generates touch input data corresponding to touched locations on the screen 202. The processing unit 204 is configured to add the touch input data to the buffer and perform gesture recognition using the input data stored in the buffer. In some embodiments, the input data may include position data representing individual touched locations within the assigned first area 206. The input data may also include individual timing data representing individual times that individual locations were touched by the user 50. When the user 50 touches any of the buttons 242, 252, 262, the generated corresponding input data may include an identification of the touched button, the timing of the touch action, and also the duration that the button was touched.

[0370] In the illustrated embodiment, the processing unit 204 is configured to use the input data to perform gesture recognition and identify a corresponding command for the performed gesture. For example, if the processing unit 204 determines from the input data that the user 50 is performing a swipe action, the processing unit 204 may then identify "move" as the command corresponding to the user's finger action. As another example, if the processing unit 204 determines from the input data that the user 50 is performing a pinch action, the processing unit 204 may then identify "reduce size" as the command corresponding to the finger action.

[0371] It should be noted that the processing unit 204 is not limited to detecting the above gestures or finger actions, and the processing unit 204 may also detect other finger actions of the user 50. As non-limiting examples, the processing unit 204 may be configured to detect one finger swipes (e.g., up / down swipes (FIG. 10A), left / right swipes (FIG. 10B)), two finger swipes, one finger touch movement (FIG. 11A), two finger touch movement (FIG. 11B), two finger pinch (FIG. 12A), two finger unpinch (FIG. 12B), two finger radial movement (FIG. 13), long press touch (FIG. 14), tap (FIG. 15), double tap, etc. In one implementation, the processing unit 204 is configured to distinguish between long press and tap actions based on the duration that the user's finger contacts the screen 202. For example, if the touch duration is 0.7 seconds or less, the processing unit 204 may determine that the finger action is a tap action. On the other hand, if the touch duration is 1 second or more, the processing unit 204 may determine that the finger action is a long press action. The processing unit 204 may also detect finger actions performed using other assigned areas of the screen 202, such as an area where the "Home" button is located, an area where the "Trigger" button is located, and an area where the "Bumper" button is located. In some implementations, the functionality of each of one or more of the "Home", "Trigger", and "Bumper" buttons described herein may correspond to the functionality of each of one or more of the "Home", "Trigger", and "Bumper" buttons as described in U.S. Patent Application No. 15 / 965,702 (incorporated herein by reference in its entirety). Similarly, in some examples, the functionality of the touchpad described herein may correspond to that described in U.S. Patent Application No. 15 / 965,702 (incorporated herein by reference in its entirety). Thus, device 200 can function as a suitable substitute for a dedicated input controller. Figure 16A shows an example of a finger action (e.g., a long press, a tap, etc.) that can be implemented on a "trigger" button and is detectable by processing unit 204.Further, the processing unit 204 may be configured to detect combinations of finger actions performed on different assigned areas of the screen 202. For example, as shown in FIG. 16B, the user 50 may press and hold the "trigger" button and, while doing so, may also perform a touch movement using the first assigned area 206 of the screen 202. The processing unit 204 may be configured to detect both of these finger actions simultaneously and consider them as a combined input for the image display device 101. For example, the processing unit 204 may determine that the above combination of finger actions is for creating a command to move the selected object. The processing unit 204 may then transmit such a command signal to the image display device to move the selected object. The processing unit 204 is configured to generate corresponding commands for the above different finger actions or combinations of finger actions performed on one or more assigned areas of the screen 202.

[0372] In some embodiments, while the user 50 is performing a finger action on the allocated area of ​​the screen 202, the user's 50 finger may unintentionally leave the screen 202. To address this situation, the processing unit 204 may be configured to determine that a finger action (e.g., a swipe action, etc.) has ended after receiving more than a predetermined threshold amount of consecutive samples indicating that the user 50 is not touching anywhere on the screen 202. For example, the processing unit 204 may wait until it receives three (or other number of) consecutive null samples before determining that such a swipe action has ended. In this example, the processing unit 204 may receive a series of touch input samples followed by two consecutive null samples (e.g., due to the user's finger not unintentionally touching the screen 202) and then a series of touch input samples. In this situation, the processing unit 204 would still treat the entire sequence of samples as if they were part of one sustained swipe action. This allows the user 50 to correct the action and end the finger action without requiring the user to re-perform the previously performed action. The above features are advantageous because they help to filter out noise and increase the system robustness for measuring or sensing errors.

[0373] In some embodiments, the processing unit 204 is configured to send the identified command to the image display device 101 (e.g., the processing module 130). The processing module 130 then determines the content to display for viewing by the user 50 based on the received command determined by the processing unit 204. In some cases, the same command may have different effects on the content displayed by the image display device 101 depending on the particular application being used by the user 50 and / or the particular content being viewed by the user 50. For example, if the user 50 is viewing a page that allows the user 50 to browse different pages of objects, a "move" command may cause the scrolling of different pages of objects. On the other hand, if the user 50 is in an application that allows objects to be moved, a "move" command may cause the object to be moved from one location to another.

[0374] As illustrated in the above embodiment, the device 200 is advantageous and improves the technical field of head-mounted image display devices in several ways. First, the device 200 can serve as a backup input device (for a head-mounted image display device) so that if a dedicated input controller for the head-mounted image display device becomes unavailable (e.g., the dedicated input controller is lost, damaged, runs out of battery, etc.), the device 200 can be used by the user 50 to input input for the head-mounted image display device instead. Many users of image display devices already carry their smartphones whenever they go out. Therefore, using a smartphone to implement the device 200 as an input controller will increase the chances that the user has an input controller for the image display device. The device may also be used as a main or sole input device for the image display device so that the image display device may not need to be accompanied by a dedicated input controller.

[0375] As mentioned, many users of image display devices already carry a smartphone or other type of mobile device whenever they go out. Therefore, implementing the apparatus 200 using a smartphone or any other type of mobile device as an input controller will improve portability and mobility for the image display device, since a user can use the image display device 101 anywhere (even without a dedicated input controller for the display device 101) as long as the user has a mobile device. Backup and auxiliary input support can also be achieved through the apparatus 200, since most users already carry a mobile device whenever they go out.

[0376] The device 200 may be used as a main device for inputting text, or alternatively, as a replacement for a dedicated input controller for inputting text. Typing text through the touch screen of the device while the user views the screen of the device through a transparent portion of the head mounted image display device is more convenient and efficient than displaying a keyboard on the viewing screen of the image display device while the user types the text using a dedicated input controller. Although smartphones are known to implement keyboards, the concept of implementing a keyboard using a smartphone in the context of providing input for a head mounted image display device is considered novel and unintuitive. This is because such a solution would require the user 50 of the head mounted image display device 101 to look through the display screen of the image display device 101 and view the screen 202 of the device 200 (on which the keyboard is displayed). Thus, the solution would require the user to shift focus from the display screen of the display image display device 101 to the screen 202 of the device 200 (visible through the display screen of the image display device 101). However, despite the shift in visual focus, it is conceivable that a touch screen keyboard implemented on the apparatus 200 as an input device for the image display device 101 may be more comfortable in some instances for some users.

[0377] Also, in embodiments where apparatus 200 is implemented using a user's mobile phone (e.g., a smartphone), apparatus 200 may also facilitate the transfer of digital images and photographs and other media content between the user's phone and image display device 101. For example, photographs taken by a camera on image display device 101 may be transferred to the user's apparatus 200, and photographs taken by a camera on apparatus 200 may be transferred to image display device 101. An apparatus that provides both content and control input for a head-mounted image display device by a user is believed to be unique.

[0378] Additionally, in some embodiments, the apparatus 200 described herein allows a user of a head-mounted image display device to access application stores and content through a network, such as the Internet, a Bluetooth network, etc. Because most smartphones already have multiple network interfaces for multiple types of network connections (e.g., Wi-Fi connections, Bluetooth connections, connections to cellular towers, etc.), implementing the apparatus 200 using a smartphone (or other type of portable network device) would have the advantage of allowing a user to obtain media content and other information from multiple sources through different types of connections. For example, the apparatus 200 implemented as an input device for the image display device 101 may also access an application store to obtain applications usable with the image display device 101. As another example, the apparatus 200 implemented as an input device for the image display device 101 may also obtain media content from the web, from another mobile device (e.g., through email, texting, airdrop, etc.). In some embodiments, the device 200 may also enable the user 50 to control account settings for and manage the image display device 101 through a user interface displayed on the screen 202 of the device 200.

[0379] Further, in some embodiments, the apparatus 200 described herein may receive notifications (for the image display device 101) from a network (e.g., the Internet) even when the head-mounted image display device 101 is not worn by the user 50. For example, the image display device 101 may push different notifications to the apparatus 200. Pushing of notifications may be performed directly by the image display device 101 or indirectly through another network device (such as a component in a cloud network). As non-limiting examples, notifications received by the apparatus 200 may be calendar notifications, advertisement notifications, social media notifications, operational notifications related to the operation of the image display device 101, etc. (e.g., battery status, storage level, update notifications, etc.), or any combination of the foregoing. Also, in some embodiments, a third party in communication with the image display device 101 may provide notifications to the image display device 101. In such cases, the image display device 101 may forward the notification to the apparatus 200 so that the user 50 may see the notification even if the user 50 is not wearing the image display device 101. The notification feature is advantageous because it allows functions such as calendar management to be more practical and timely reminders and real-time updates for the image display device 101 may be received by the user via the apparatus 200.

[0380] Sensory feedback

[0381] In some embodiments, the processing unit 204 may be configured to optionally operate a feedback component in the device 200 in response to the finger action of the user 50. Such a feature is advantageous in that it allows the user 50 to know that his / her finger has reached or crossed the boundary of the first area 206. In particular, the user 50 may not be viewing the device 200 when the user 50 is viewing the content displayed by the image display device 101. Thus, the user 50 may not visually notice that his / her finger has moved outside the boundary of the first area 206 (assigned to receive finger action input). The feedback feature solves this technical problem and provides a technical improvement for the image display system 100 by informing the user 50 of the image display system 100 via feedback that he / she is about to cross, has reached, or has crossed the boundary of the assigned first area 206. The feedback may be in the form of vibration (tactile feedback), which may include one or more mechanical pulses. Alternatively, or in addition, the feedback may include audio feedback.

[0382] As shown in FIG. 17, in some embodiments, the first area 206 may have a boundary 280, and the processing unit 204 may be configured to operate the feedback component when the finger of the user 50 crosses, reaches, or moves to a location within a predefined distance 282 of the boundary 280. In one implementation, the first assigned area 206 has a square or rectangular shape with four boundaries 280. In such a case, the processing unit 204 may be configured to operate the feedback component when the finger of the user 50 crosses, reaches, or moves to a location within a predefined distance from any of the four boundaries 280 of the first assigned area 206. The feedback component may include a tactile feedback component, a speaker, any of the other types of components capable of generating a feedback signal (such as a visual signal for display by the image display device 101), or any combination of the foregoing. Thus, as used herein, the term "feedback component" may include one or more components for providing one or more sensory feedback. Similarly, as used herein, the term "feedback" may include one or more different types of feedback.

[0383] As another example, the first area 206 may have one or more boundaries 280 that at least partially surround the reference location. For example, the first area 206 may have a square or rectangular shape that surrounds the reference location (e.g., the center), or may have a circular shape that surrounds the reference location (e.g., the center). In such a case, the processing unit 204 may be configured to operate a feedback component within the device 200 in response to the finger of the user 50 reaching a predefined distance from the reference location.

[0384] Also, in some embodiments, the processing unit 204 may be configured to operate a feedback component in the device 200 to generate different types of feedback based on different individual spatial relationships between one or more fingers of the user 50 relative to the first area 206. This allows the user 50 to know the extent to which their fingers have reached one or more boundaries 280 of the assigned first area 206. In one example, the different individual spatial relationships may be that one or a point between two of the one or more fingers reaches a different individual distance 282 from the boundary 280 of the assigned first area 206. In another example, the different individual spatial relationships may be different distances (above a threshold) between (1) one or a point between two of the user's one or more fingers and (2) a reference location within the assigned first area 206.

[0385] In one embodiment, the different types of feedback may include a first tactile impulse with a first amplitude and a second tactile impulse with a second amplitude different from the first amplitude. In such a case, the amplitude of the tactile impulse would increase as the user's finger moves closer to the boundary 280 of the assigned first area 206 (or moves further away from a reference location bounded by one or more boundaries 280).

[0386] In another example, the different types of feedback include a first number of tactile impulses and a second number of tactile impulses different from the first number. In such a case, the number of tactile impulses would increase as the user's finger moves closer to the boundary 280 of the assigned first area 206 (or moves further away from a reference location bounded by one or more boundaries 280).

[0387] In a further example, the different types of feedback include a first series of tactile impulses with a first frequency and a second series of tactile impulses with a second frequency different from the first frequency. In such a case, the frequency of the tactile impulses would increase as the user's finger moves closer to the boundary 280 of the assigned first area 206 (or moves further away from a reference location bounded by one or more boundaries 280).

[0388] In another embodiment, the different types of feedback may include a first audio signal accompanied by a first tone and a second audio signal accompanied by a second tone.

[0389] In yet another embodiment, the different types of feedback may include a first number of audio signals and a second number of audio signals, where the first number is different from the second number.

[0390] In a further embodiment, the different types of feedback may include a first audio message and a second audio message that is different from the first audio message.

[0391] In other embodiments, the processing unit 204 may operate the feedback component and generate different types of feedback based on the user's fingers reaching different boundaries 280 of the assigned first area 206. For example, the assigned first area 206 may have a first boundary 280 (e.g., a left boundary or a right boundary) and a second boundary 280 (e.g., an upper boundary or a lower boundary). In such a case, the processing unit 204 may be configured to operate the feedback component and generate a first type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the first boundary, and to operate the feedback component and generate a second type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance from the second boundary. Such a feature may be desirable because it allows the user 50 to know which boundary 280 their fingers are approaching without looking at the device 200.

[0392] Also, in other embodiments, the processing unit 204 may be configured to operate the feedback component based on the swipe direction of the user's finger. For example, assume that the assigned first area 206 has a rectangular shape with a top boundary, a bottom boundary, a left boundary, and a right boundary. If the user 50 swipes his / her finger upwards, the boundary 280 that is likely to be reached will be the top boundary. In such a case, the processing unit 204 may then operate the feedback component and generate a type of feedback to inform the user 50 that the user's finger is reaching the top boundary.

[0393] In some embodiments, it may be desirable to ensure that feedback is provided only in situations where a touch outside (or close to) the boundary 280 of the first area 206 originates from the user 50 using the assigned first area 206. For example, if the user 50 touches outside (or within a predefined distance from) the assigned first area 206 and the touch action is not part of a sustained swipe action starting from within the first area 206, the processing unit 204 may not operate the feedback component and provide any feedback. On the other hand, if the touch outside (or within a predefined distance from) the assigned first area 206 is part of a swipe action starting from within the assigned first area 206, the processing unit 204 may operate the feedback component and provide feedback.

[0394] FIG. 18 illustrates an algorithm or method 300 for providing feedback that considers the above two scenarios. First, the processing unit 204 determines whether an input signal is present indicating receipt of a touch input by the user 50 (item 302). The input signal may be generated as a result of the user 50 touching anywhere within the touch-sensitive area of ​​the device 200. If such an input signal is not present, the processing unit 204 then clears a flag (item 304). In the illustrated embodiment, the flag is used to track whether the user 50 initiated a finger action by touching a location inside the assigned first area 206. If the user initiated such an action, the flag will be set. If not, the flag will be cleared. After the flag is cleared, the processing unit 204 continues to determine whether the user 50 touched a portion of any touch-sensitive area of ​​the device 200 (e.g., areas outside and within the assigned first area 206) such that the method loops back to item 302.

[0395] If the processing unit 204 determines that an input signal is present indicating receipt of a touch input by the user 50, the processing unit 204 then determines whether the touched location is within the assigned first area 206 of the screen 202 (item 306). If the touched location is inside the assigned first area 206, the processing unit 204 sets a flag to indicate that a finger action occurred involving the user 50 touching a location within the assigned first area 206 (item 308). The processing unit 204 also adds the touch input data to a buffer (item 310) and performs gesture recognition using the input data stored in the buffer (item 312). As discussed, the buffer is configured to store input data generated by the user 50 performing finger actions on the assigned first area 206 and / or other assigned areas. In some embodiments, in item 312, the processing unit 204 performs gesture recognition using the input data and identifies a corresponding command related to the performed gesture (finger action). The processing unit 204 also generates or identifies a corresponding command for the detected finger action to be performed on one or more assigned areas of the screen 202. In some embodiments, the processing unit 204 is configured to send the command to the image display device 101 (e.g., the processing module 130). The processing module 130 then determines content to be displayed for viewing by the user 50 based on the received command provided by the apparatus 200.

[0396] Returning to FIG. 18 , on the other hand, if the processing unit 204 determines in item 306 that the touched location is outside the assigned first area 206, the processing unit 204 then determines whether the flag was previously set (item 316). If the flag was not previously set, this means that the currently touched location is not part of a finger action that started from a location within the assigned first area 206 (e.g., the currently touched location may result from a tap, grip, etc. of the user 50 at a location outside the assigned first area 206). In such a case, the processing unit 204 does not operate the feedback component, and no feedback is provided to the user 50. The method then loops back to item 302 so that the processing unit 204 may continue to determine whether the user 50 has touched any portion of the touch-sensitive region of the device 200 (e.g., areas outside and within the assigned first area 206).

[0397] On the other hand, if the processing unit 204 determines in item 316 that the flag was previously set, this means that the currently touched location (outside the assigned first area 206 or within a predefined distance 282 from the boundary 280 of the assigned first area 206) is part of a trajectory of a finger action that started from within the first area 206. In such a case, the processing unit 204 then operates a feedback component to provide feedback (item 318). Once the feedback is provided, the processing unit may then flush a buffer that stores input data generated from the user 50 touching the assigned first area 206 (item 314). Flushing the buffer means that the previous input data resulting from the user 50 touching the assigned first area 206 is deleted and commands will not be identified and generated because the user's finger has reached or exceeded the boundary 280 of the assigned first area 206.

[0398] CHANGE OF CONTENT BASED ON FINGER-GENERATED INPUT SIGNALS AND / OR SENSOR SIGNALS

[0399] As described above, the assigned first area 206 (functioning like a touchpad) and other assigned areas (like areas 240, 250, 260) of the screen 202 enable the user 50 of the video display device 101 to provide various finger-generated inputs for the video display device 101. The processing unit 204 of the apparatus 200 is configured to generate electronic signals (resulting from the finger-generated inputs) for causing a change in the content displayed by the image display device 101. In some cases, the electronic signals may represent an identification of a finger action. Alternatively, or in addition, the electronic signals may represent a command or an identification of a command determined by the processing unit 204 based on recognition of the finger action inputs generated using the assigned areas 206, 240, 250, 260 (like those described with reference to FIGS. 10-16 and items 312 and 314 of FIG. 18). In some embodiments, the apparatus 200 is configured to transmit such electronic signals to the image display device 101 via a wired connection. In other embodiments, apparatus 200 may use a wireless communication module to convert the electronic signals into wireless form for wireless transmission to image display device 101. Processing module 130 of video display device 101 receives the signals from apparatus 200 and alters content for display by display subsystem 110 based on the signals.

[0400] As non-limiting examples, the change in content may be a change in size of the content, a change in position of the content, a change in shape of the content, a change in color of the content, a replacement of information in the content, an increase or decrease in the amount of information in the content, or any combination of the foregoing. In some cases, the change in content caused by the finger-generated input (generated using the assigned first area 206) may be a movement of a cursor or pointer. In other cases, the change in content caused by the finger-generated input may be a change in the selection of an item to another selection of another item displayed by the image display device 101. In still other cases, the change in content caused by the finger-generated input may be a movement of an object (e.g., a photo, a computer-generated image, a cartoon, etc.) across the display of the image display device 101, the movement occurring within the viewing plane of the user 50 (e.g., the XY plane) or along the viewing depth of the user 50 (e.g., along the Z-axis). Also, in some cases, the change in size of the content may be a change in the size of the object (e.g., a photo, a computer-generated image, a cartoon, etc.) displayed by the image display device 101.

[0401] It should be noted that finger actions detected by the processing unit 204 may be used by the processing unit 204 to determine a variety of different commands (examples of electronic signals), which the processing unit 204 may transmit (wirelessly or through a wired connection) to the image display device 101 to enable the image display device 101 to change the displayed content based on the commands. As a non-limiting example, commands determined from detected touch movements (such as those shown in FIG. 11A ) may be processed by the image display device 101 to target elements, control cursor movement, navigate within the plane of the viewing screen of the image display device 101, etc. Commands determined from detected upward or downward swipes (such as those shown in FIG. 10A ) may be processed by the image display device 101 to scroll content in simple grid and list layouts. Commands determined from a detected left or right swipe (such as that shown in FIG. 10B) may be processed by the image display device 101 to move to a next page, move to a different section, etc., relative to a current viewing frame for display by the image display device 101. Commands determined from a two finger touch and scroll action (such as that shown in FIG. 11B) may be processed by the image display device 101 to provide inertial scrolling of content displayed by the image display device 101 (i.e., if the two finger touch is moved slowly, the page scrolls slowly and precisely, and if the two finger touch is moved with a rapid swipe, the page is turned over in one go). Commands determined from a pinch action (such as that shown in FIG. 12A) or an unpinch action (such as that shown in FIG. 12B) may be processed by the image display device 101 to change the size of an object displayed by the image display device 101.A command determined from a detected long press (such as that shown in FIG. 14) within the assigned first area 206 may be processed by the image display device 101 to open a menu, open a browser, open an application, etc. for display by the image display device 101. A command determined from a detected tap action (such as that shown in FIG. 15) within the assigned first area 206 or a tap action on a "trigger" button (such as that shown in FIG. 16B) may be processed by the image display device 101 to open an additional function menu, etc. for display by the image display device 101. The above functions resulting from commands generated by the processing unit 204 are only examples. In other embodiments, the same detected finger actions described above may be used to generate other different commands for implementing other functions different from the examples described.

[0402] Also, in some embodiments, a sequence of commands determined from the sequence of detected finger actions may be transmitted from the processing unit 204 to the image display device 101 to enable the user 50 to perform various tasks on the content displayed by the image display device 101. For example, a sequence of commands may be provided by the processing unit 204 for 3D placement of objects and content extraction. In one implementation, the user 50 presses and holds the "trigger button", grasps an object displayed by the image display device 101, then performs a touch movement using the first assigned area 206 to move the grasped object, and then releases the "trigger button" to place the object at the desired location. In some embodiments, the object is grasped as long as the "trigger" button is pressed and is dropped as soon as the "trigger" button is released. In such a case, the user 50 may use one hand to press the "trigger" button while the other hand is used to move the grasped object. Alternatively, after the object is grasped, the user can release the "trigger" button and the object will still remain grasped. In such a case, the user can use the same hand to move the grasped object. After the grasped object is desirably placed, the user can then tap or long press the "trigger" button and release the grasped object. The 3D placement feature may enable the user 50 to move any object displayed by the image display device 101. For example, the user 50 may use the above features to drag and drop images into an application, to compose a slide show, to move a web slider, etc.In some embodiments, while the object is being grasped, the user 50 can use the assigned first area 206 and / or head pose to move the grasped object, can use the assigned first area 206 to rotate the grasped object (using a radial movement of two fingers such as that shown in FIG. 13), and can use the assigned first area 206 to scale the grasped object (using a pinch or unpinch action such as that shown in FIGS. 12A / 12B).

[0403] As another example, the processing unit 204 may detect a swipe action by the user 50 and, in response, provide a text box to allow the user 50 to type text (e.g., English characters, phonetic or ideographic characters in other languages, numbers, punctuation marks, special characters, symbols, emojis, text graphics, etc.). This swipe / type feature may provide a fast and convenient way for the user 50 to type text and other information.

[0404] In some embodiments, in addition to the input signal generated using the finger action performed on the assigned area 206, the processing unit 204 of the device 200 may also obtain a sensor signal associated with the orientation of the device 200 and use the combination of the input signal and the sensor signal to cause a change in the content displayed by the image display device 101. In particular, the processing unit 204 may determine a command based on the combination of the input signal and the sensor signal and may transmit such a command to the image display device 101. The image display device 101 then processes the command and changes the content based on the command. Alternatively, the processing unit 204 may determine a first command for the input signal and a second command for the sensor signal and transmit both commands to the image display device 101. The image display device 101 then processes the command and changes the content based on both commands.

[0405] For example, in some embodiments, device 200 further includes an orientation sensor for sensing an orientation of device 200. The orientation sensor may be either an inertial measurement unit (IMU) or other type of orientation sensor. In such a case, processing unit 204 may be configured to generate electronic signals to vary the content displayed by image display device 101 based on both (1) the finger-generated input signal and (2) the sensed orientation of the device.

[0406] In some cases, the finger-generated input signal may be associated with a pinch or unpinch action of the user 50. In such cases, the device, i.e., the processing unit 204, may be configured to generate electronic signals for changing the content displayed by the image display device 101 based on (1) the input signal associated with the pinch or unpinch action and (2) the sensed orientation of the device 200. For example, the processing unit 204 may be configured to (1) generate electronic signals for contracting or expanding the content in a first plane if a pinch or unpinch action is sensed by the device 200 while the device 200 is in a first orientation, and (2) generate electronic signals for contracting or expanding the content in a second plane if a pinch or unpinch action is sensed by the device 200 while the device 200 is in a second orientation different from the first orientation, the second plane being different from the first plane. In some embodiments, device 200 may be considered to have a first orientation when the long axis of device 200 forms an angle with a horizontal plane that is less than 45°, or more preferably, less than 30°, and device 200 may be considered to have a second orientation when the long axis of device 200 forms an angle with a vertical plane that is less than 45°, or more preferably, less than 30°.

[0407] Additionally, a first plane in which the content contracts or expands due to the device being in a first orientation may be perpendicular to a second plane in which the content contracts or expands due to the device being in a second orientation. For example, the first plane may be a YZ plane in the virtual three-dimensional environment and the second plane includes an XY plane in the virtual three-dimensional environment.

[0408] In other embodiments, instead of a pinch or unpinch action of user 50, the input signal may result from other types of finger-generated actions. For example, the input signal may result from a swipe action performed by user 50. In such cases, the apparatus, i.e., processing unit 204, may be configured to generate an electronic signal to change the content displayed by image display device 101 based on (1) the input signal associated with the swipe action and (2) the sensed orientation of apparatus 200.

[0409] For example, the processing unit 204 may be configured to (1) generate an electronic signal to move the content in a first plane if a pinch or unpinch action is sensed by the device 200 while the device 200 is in a first orientation, and (2) generate an electronic signal to move the content in a second plane if a pinch or unpinch action is sensed by the device 200 while the device 200 is in a second orientation different from the first orientation, the second plane being different from the first plane. In some embodiments, the device 200 may be considered to have a first orientation when the long axis of the device 200 forms an angle with a horizontal plane that is less than 45°, or more preferably, less than 30°. Also, the device 200 may be considered to have a second orientation when the long axis of the device 200 forms an angle with a vertical plane that is less than 45°, or more preferably, less than 30°.

[0410] Additionally, a first plane in which content moves due to the device being in a first orientation may be perpendicular to a second plane in which content moves due to the device being in a second orientation. For example, the first plane may be a YZ plane in the virtual three-dimensional environment and the second plane includes an XY plane in the virtual three-dimensional environment.

[0411] In some embodiments, the content is in a virtual three-dimensional environment and the processing unit 204 is configured to generate electronic signals to change the content displayed by the image display device 101 by moving the content closer to or farther from the user 50 when a swipe action is sensed by the device 200 while an orientation of the device 200 is approximately parallel to the horizontal plane (e.g., forming an angle of ±30° from the horizontal plane). The processing unit 204 is also configured to generate electronic signals to move the content displayed by the image display device 101 in a vertical plane within the three-dimensional environment when a swipe action is sensed by the device 200 while an orientation of the device 200 is approximately perpendicular to the horizontal plane (e.g., forming an angle of ±30° from the vertical plane).

[0412] In some embodiments, the processing unit 204 is configured to generate an electronic signal to cause the content to expand in one or more directions based on the sensed orientation of the device 200. For example, when the device 200 is in a first orientation (e.g., an upright orientation), the electronic signal may command the content to expand in a first direction corresponding to the first orientation of the device 200. When the device 200 is in a second orientation (e.g., a face-up orientation), the electronic signal may command the content to expand in a second direction corresponding to the second orientation of the device 200.

[0413] Also, in some embodiments, the processing unit 204 is configured to generate electronic signals to move or rotate content based on the sensed orientation of the device 200.

[0414] Additionally, in some embodiments, the device 200 further includes a movement sensor for sensing movement of the device 200, and the processing unit 204 is configured to generate an electronic signal for changing (e.g., moving) content displayed by the image display device 101 based on the sensed movement of the device 200. The movement sensor may include an accelerometer or any other component that may detect movement and / or direction of movement.

[0415] Figures 19A and 19B illustrate examples of some of the features described above. In particular, Figure 19A illustrates examples of finger actions and corresponding effects on the displayed content that may be performed on the device of Figure 6 while the device is in a face-up orientation. Figure 19B illustrates examples of finger actions and corresponding effects on the displayed content that may be performed on the device of Figure 6 while the device is in an upright orientation. As shown in FIG. 19A, when the device is held by the user 50 in a table-up orientation, the user 50 may perform (1) a radial movement 400a of two fingers on the assigned first area 206 of the screen 202 of the device 200 to rotate the object 402 displayed by the image display device 101 about the Y-axis, (2) a vertical swipe movement 400b on the assigned first area 206 of the screen 202 of the device 200 to move the object 402 further apart along the Z-axis, and (3) an unpinch movement 400c on the assigned first area 206 of the screen 202 of the device 200 to increase the size of the object 402 along the Z-axis. On the other hand, as shown in Fig. 19B, when the device is held by the user 50 in an upright orientation, the user 50 may perform (1) a radial movement 410a of two fingers on the assigned first area 206 of the screen 202 of the device 200 to cause the object 402 to be displayed by the image display device 101 to rotate about the Z-axis, (2) a vertical swipe movement 410b on the assigned first area 206 of the screen 202 of the device 200 to move the object 402 up (or down) along the Y-axis, and (3) an unpinch movement 410c on the assigned first area 206 of the screen 202 of the device 200 to increase the size of the object 402 along the Y-axis. Thus, depending on the orientation of the device 200, the same finger action will achieve different effects on the content displayed by the image display device 101.

[0416] In some embodiments, the processing unit 204 may be configured to detect six different scenarios of finger action input and / or orientation sensor signals and generate corresponding control signals to move an object displayed by the image display device 101 in six different degrees of freedom, respectively. For example, based on the finger action input and / or orientation sensor signals, the object displayed by the image display device 101 may be translated along the X-axis, translated along the Y-axis, translated along the Z-axis, rotated about the X-axis, rotated about the Y-axis, and rotated about the Z-axis.

[0417] In some embodiments, the electronic signal generated by the processing unit 204 of the device 200 may also cause the image display device 101 to display a touch hint so that the user 50 viewing the screen of the image display device 101 may confirm the finger action being detected by the processing unit 204. For example, when the user 50 performs a two-finger touch movement on the first assigned area 206 of the screen 202 and inertially moves an object displayed by the image display device 101, the electronic signal generated by the processing unit 204 based on the detection of the two-finger touch movement may be transmitted to the image display device 101, which then displays a graphic (touch hint) to inform the user 50 that he is performing a two-finger touch movement. The same feature may be applied to other types of detected finger actions. Thus, the image display device 101 may display different finger action indicators to inform the user 50 that different individual finger actions are being detected by the device 200.

[0418] It should be noted that, in one or more embodiments, any feature described herein may be implemented by the processing unit 204 of the apparatus 200 and / or the processing unit 130 of the image display device 101. Thus, one or more features described herein as being implemented by the processing unit 204 of the apparatus 200 may alternatively be implemented by the processing unit 130 of the image display device 101, or by a combination of the processing unit 204 of the apparatus 200 and the processing unit 130 of the image display device 101. Similarly, one or more features described herein as being implemented by the processing unit 130 of the image display device 101 may alternatively be implemented by the processing unit 204 of the apparatus 200, or by a combination of the processing unit 204 of the apparatus 200 and the processing unit 130 of the image display device 101.

[0419] (Gesture Handoff)

[0420] In some embodiments, the image display device 101 may include a camera for detecting hand gestures of the user 50, and the processing unit 130 of the image display device 101 may interpret the detected gestures and generate corresponding control signals for operating the image display device 101. In some embodiments, gestures performed by the user 50 on the allocated first area 206 may be combined with gestures detected by the camera of the image display device 101 to achieve a desired operation. For example, in some embodiments, the device 200 may display content on its screen, and the user 50 may perform a pinch action on the screen of the device 200 to extract the displayed content. The processing unit 204 detects the pinch action performed using the device 200 and interprets it as a control signal for extracting the content. The user 50 may then perform an unpinch action within the field of view of the camera of the image display device 101. Image display device 101 detects the pinch-unpinch action and interprets it as a control signal to place the content at a location presented by (or viewable through) the screen of image display device 101. Thus, gesture detection performed by apparatus 200 may be combined with gesture detection performed by image display device 101 to affect content displayed by apparatus 200 and / or content displayed by image display device 101.

[0421] In another embodiment, the image display device 101 may display content on its screen, and the user 50 may perform a pinch action in front of the camera of the image display device 101 to extract the displayed content. The processing unit 130 of the image display device 101 detects the pinch action as captured by the camera and interprets it as a control signal to extract the content. The user 50 may then perform an unpinch action on the device 200. The device 200 detects the unpinch action and interprets it as a control signal to place the content in a location presented by the screen of the device 200.

[0422] Also, in some embodiments, content displayed by apparatus 200 may be moved to the "environment" of image display device 101 by selecting the content on apparatus 200 and moving the content to an edge of the display area of ​​apparatus 200 (e.g., by touch movement, swiping, etc.). When processing unit 204 detects that the content has been moved to an edge of the display, processing unit 204 then determines that the content should be moved to the display screen of image display device 101. Apparatus 200 then transmits a control signal to cause the content to be displayed by image display device 101. After the content has been "moved" from the screen of apparatus 200 to the screen of image display device 101, user 50 may then perform further actions on the content using apparatus 200 (e.g., acting on the assigned first area 206 to move the content, resize the content, etc.) and / or by using hand gestures for detection by the camera of image display device 101.

[0423] In some embodiments, the "moving" of content from the screen of apparatus 200 to the screen of image display device 101 (or vice versa) may be performed to provide temporal continuity. For example, as soon as the content disappears on the screen of apparatus 200, processing unit 130 of image display device 101 may immediately generate and provide an image of the content for display by the screen of image display device 101. In other embodiments, the "moving" of content from the screen of apparatus 200 to the screen of image display device 101 (or vice versa) may be performed without any temporal continuity. For example, there may be a delay to provide an image of the content for display by the screen of image display device 101 after the content disappears on the screen of apparatus 200.

[0424] Also, in some embodiments, the "moving" of content from the screen of device 200 to the screen of image display device 101 (or vice versa) may be performed to provide spatial continuity. For example, if content on the screen of device 200 is moved to the right edge of the display to move the content to the screen of image display device 101, as soon as the content is moved off the screen of device 200, image display device 101 may immediately display the content next to the left edge of the screen of image display device 101. Similarly, as another example, if content on the screen of image display device 101 is moved to the right edge of the display to move the content to the screen of device 200, as soon as the content is moved off the screen of image display device 101, device 200 may immediately display the content next to the left edge of the screen of device 200.

[0425] In other embodiments, the "movement" of content from the screen of device 200 to the screen of image display device 101 (or vice versa) may be performed without any spatial continuity. For example, after content on the screen of device 200 is moved off-screen, the content may always appear in a predefined location (e.g., center) on the screen of image display device 101.

[0426] In one use example, the above features may be implemented to enable user 50 to deal cards in a game of poker. For example, user 50 may use device 200 to perform a swipe action to deal poker cards. The poker cards may move off the screen of device 200 and appear on the screen of image display device 101 (or on the screen of another image display device used by another user, which may communicate with image display device 101 via a network such as the Internet, Wi-Fi, etc.).

[0427] Also, in some embodiments, the camera of the image display device 101 for detecting user gestures may be used to view the user's hand while the user 50 is performing finger action inputs on the device 200. In such a case, the processing unit 130 of the image display device 101 may detect finger actions by the user 50 and match the detection of the finger gesture detected by the device 200.

[0428] Additionally, in some embodiments, the image display device 101 is configured to detect whether the user 50 is using the device 200. When the user 50 is using the device 200 to type input, the image display device 101 may disable its gesture detection camera. Alternatively, in some examples, when the user 50 is using the device 200 to type input, the image display device 101 may keep its camera enabled and simply refrain from performing one or more gesture detection processes on the image captured by the camera. As soon as the user 50 lifts his / her finger away from the screen of the device 200, the image display device 101 may then activate the gesture detection camera and allow the camera to take over the gesture detection function. In one implementation, the processing unit 130 of the image display device 101 is configured to disable the gesture detection camera of the image display device 101 as long as the device 200 receives input generated by the finger action of the user 50. The processing unit 130 may be configured to receive a signal from the device 200 as soon as the user 50 lifts their finger away from the screen of the device 200. In response to such a signal, the processing unit 130 then activates the gesture detection camera of the image display device 101 and enables the gesture detected by the camera to be used as an input image for the display device 101.

[0429] (Methods implemented by the processing unit and / or applications within the processing unit) law)

[0430] FIG. 20 illustrates a method 500 according to some embodiments. The method 500 may be implemented by the processing unit 204 and / or an application in the processing unit 204 of the device 200. The method 500 includes allocating a first area 206 of the screen 202 of the device 200 for sensing a finger action of a user 50 of the image display device 101, the image display device 101 being configured for head-worn by the user, and the device 200 being different from the image display device 101 (item 502). The image display device 101 may be any of those shown in FIGS. 1-4. The method 500 also includes generating an electronic signal for causing a change in content displayed by the image display device 101 based on the user's finger action sensed by the allocated first area 206 of the screen 202 of the device 101 (item 504).

[0431] In some embodiments, the electronic signal may be a command determined by the processing unit 204 based on gesture recognition or by an application in the processing unit 204. The electronic signal may be transmitted by the apparatus 200 to the processing unit 130 of the image display device 101, which then changes the content based on the electronic signal. Alternatively, if the apparatus 200 is involved in providing content for display by the image display device 101, the processing unit 204 may change the content being displayed based on the electronic signal it provides. Thus, the electronic signal may directly or indirectly cause a change in the content displayed by the image display device 101.

[0432] Optionally, in the method 500 , the screen 202 has a touch-sensitive area 210 and the allocated first area 206 is less than a total area of ​​the touch-sensitive area 210 .

[0433] Optionally, the method 500 may further include ignoring input generated by the user 50 using a portion of the touch-sensitive region 210 that is not part of the assigned first area 206 and that is not part of the assigned button.

[0434] Optionally, the method 500 may further include generating a control signal for operating a feedback component in response to a finger action of the user 50 .

[0435] Optionally, in method 500, the first area has a boundary and the finger action of user 50 includes moving the user's finger to a location that crosses, reaches, or is within a predefined distance of the boundary.

[0436] Optionally, in method 500, the first area 206 has one or more boundaries at least partially surrounding the reference location, and the control signal is for operating the feedback component in response to the finger of the user 50 reaching a predefined distance from the reference location.

[0437] Optionally, method 500 further includes generating different control signals for operating the feedback components to generate different respective types of feedback based on different respective spatial relationships between one or more fingers of user 50 relative to first area 206.

[0438] Optionally, in method 500, the different types of feedback include a first tactile impulse with a first amplitude and a second tactile impulse with a second amplitude different from the first amplitude.

[0439] Optionally, in method 500, the different types of feedback include a first number of tactile impulses and a second number of tactile impulses different from the first number.

[0440] Optionally, in method 500, the different types of feedback include a first series of tactile impulses with a first frequency and a second series of tactile impulses with a second frequency different from the first frequency.

[0441] Optionally, in method 500, the different respective spatial relationships include different distances between (1) one of the user's one or more fingers, or a point between two fingers, and (2) a reference location within the assigned first area 206.

[0442] Optionally, in the method 500 , the reference location includes the center of the assigned first area 206 .

[0443] Optionally, in method 500, the difference distance exceeds a threshold.

[0444] Optionally, in method 500, the different respective spatial relationships include one of the one or more fingers, or a point between two fingers, reaching different respective distances from a boundary of the assigned first area 206.

[0445] Optionally, in method 500, the assigned first area 206 has a first boundary and a second boundary, the different types of feedback include at least a first type of feedback and a second type of feedback, and the feedback component is operated to generate the first type of feedback when one or more fingers of the user 50 cross, reach, or move to a location within a predefined distance from the first boundary, and the feedback component is operated to generate the second type of feedback when one or more fingers of the user 50 cross, reach, or move to a location within a predefined distance from the second boundary.

[0446] Optionally, in the method 500, the first boundary includes a left or right boundary of the allocated first area 206 and the second boundary includes an upper or lower boundary.

[0447] Optionally, in method 500, the control signal for operating the feedback component is based on the swipe direction.

[0448] Optionally, the method 500 further includes receiving an input signal associated with a pinch or unpinch action performed by the user 50 on the allocated first area 210 of the screen 202 .

[0449] Optionally, in the method 500, the electronic signal is for varying a size of content displayed by the image display device 101 in response to an input signal associated with a pinch or unpinch action.

[0450] Optionally, the method 500 further includes obtaining an orientation of the device 200 from an orientation sensor, and the electronic signal is for changing the content displayed by the image display device 101 based on the input signal associated with the pinch or unpinch action and the orientation of the device 200.

[0451] Optionally, in method 500, the content is changed by contracting or expanding the content in a first plane when a pinch or unpinch action is sensed by the assigned first area 206 while the device 200 is in a first orientation, and the content is changed by contracting or expanding the content in a second plane when a pinch or unpinch action is sensed by the assigned first area 206 while the device 200 is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0452] Optionally, in method 500, device 200 has a first orientation when a long axis of the device forms an angle of less than 45° with a horizontal plane.

[0453] Optionally, in method 500, device 200 has a second orientation when a long axis of the device forms an angle of less than 45° with a vertical plane.

[0454] Optionally, in the method 500, the first plane includes a YZ plane in the virtual three-dimensional environment and the second plane includes an XY plane in the virtual three-dimensional environment.

[0455] Optionally, in the method 500, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0456] Optionally, in the method 500, the first plane is perpendicular to the second plane.

[0457] Optionally, the method 500 further includes receiving an input signal associated with a swipe action performed by the user 50 on the allocated first area 206 of the screen 202 .

[0458] Optionally, in method 500, the electronic signal is for moving content displayed by the image display device 101 in response to the detected swipe action.

[0459] Optionally, the method 500 further includes a step of obtaining an orientation of the device 200 from an orientation sensor, and the electronic signal is for changing the content displayed by the image display device 101 based on the input signal associated with the swipe action and the orientation of the device 200.

[0460] Optionally, in the method 500, the content is changed by moving the content in a first plane when a swipe action is sensed by the assigned first area 206 while the device 200 is in a first orientation, and the content is changed by moving the content in a second plane when a swipe action is sensed by the assigned first area 206 while the device 200 is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0461] Optionally, in method 500, device 200 has a first orientation when a long axis of device 200 forms an angle of less than 45° with a horizontal plane.

[0462] Optionally, in method 500, device 200 has a second orientation when a long axis of device 200 forms an angle of less than 45° with a vertical plane.

[0463] Optionally, in the method 500, the first plane includes a YZ plane in the virtual three-dimensional environment and the second plane includes an XY plane in the virtual three-dimensional environment.

[0464] Optionally, in the method 500, the first plane and the second plane are relative to a virtual three-dimensional environment.

[0465] Optionally, in the method 500, the first plane is perpendicular to the second plane.

[0466] Optionally, in method 500, the content is within a virtual three-dimensional environment and the control signal is for moving the content displayed by the image display device 101 closer to or farther away from the user when a swipe action is sensed by the assigned first area 206 while the orientation of the device 200 is approximately parallel to a horizontal plane.

[0467] Optionally, in method 500, the content is within a virtual three-dimensional environment and the control signal is for moving the content displayed by the image display device 101 in a vertical plane within the three-dimensional environment when a swipe action is sensed by the assigned first area 206 while an orientation of the device 200 is substantially perpendicular to a horizontal plane.

[0468] Optionally, the method 500 further includes obtaining a sensor input indicative of a sensed orientation of the device 200, and the electronic signal is for varying the content displayed by the image display device 101 based on the sensor input indicative of the sensed orientation of the device 200.

[0469] Optionally, in method 500, the control signal is for altering the content by expanding the content in one or more directions based on a sensor input indicative of a sensed orientation of device 200.

[0470] Optionally, in method 500 , the control signal is for altering the content by rotating the content based on a sensor input indicative of a sensed orientation of device 200 .

[0471] Optionally, in method 500 , the control signal is for changing the content by moving the content based on a sensor input indicative of a sensed orientation of device 200 .

[0472] Optionally, the method 500 further includes obtaining a sensor input indicative of a sensed movement of the apparatus 200, and the control signal is for changing the content displayed by the image display device 101 based on the sensor input indicative of the sensed movement of the apparatus 200.

[0473] Optionally, in method 500 , the electronic signal is for altering the content by moving the content based on a sensor input indicative of a sensed movement of device 200 .

[0474] Optionally, in the method 500, the device 200 is a handheld device.

[0475] Optionally, the handheld device includes a mobile phone, a smart phone, a personal digital assistant (PDA), or a tablet.

[0476] Optionally, in the method 500 , the allocated first area 206 of the screen 202 does not display the object while the allocated first area 206 of the screen 202 is sensitive to a finger action of the user 50 .

[0477] Optionally, the method 500 further includes operating the screen 202 to display the grid of dots within the allocated first area 206 of the screen 202 .

[0478] Optionally, method 500 further includes changing a characteristic of one or more of the dots in response to a user 50 touching a portion of the assigned first area 206 of the screen 202 in which one or more of the dots are displayed.

[0479] Optionally, the method 500 further comprises assigning a second area 202 of the screen as the first button.

[0480] Optionally, in the method 500, the first button is a "home" button.

[0481] Optionally, the method 500 further includes assigning a third area of ​​the screen 202 as a second button.

[0482] Optionally, in the method 500, the second button is a "toggle" button.

[0483] Optionally, the method 500 further includes assigning a fourth area of ​​the screen 202 as a third button.

[0484] Optionally, in the method 500, the third button is a "bumper" button.

[0485] Optionally, method 500 further includes, in response to user 50 touching the assigned second area of ​​the screen, assigning a second area of ​​screen 202 as a keyboard activation button and operating screen 202 to display the keyboard.

[0486] Optionally, the method 500 further comprises the step of wirelessly communicating with the image display device 101 .

[0487] Optionally, the method 500 further comprises communicating with the image display device 101 via a cable.

[0488] Optionally, in the method 500, the device 200 comprises a non-transitory medium that stores instructions, and the act of allocating the first area 206 of the screen 202 to sense finger actions of the user 50 is performed based on the instructions.

[0489] Optionally, in method 500, changing the content includes changing a size of the content, changing a position of the content, changing a shape of the content, changing a color of the content, replacing information within the content, increasing or decreasing the amount of information within the content, or any combination of the above.

[0490] In some embodiments, method 500 may further include detecting a gesture or finger action, such as a one finger touch movement, a two finger touch movement, a one finger swipe, a two finger swipe, a two finger pinch, a two finger unpinch, a two finger radial movement, a tap, a double tap, or any combination of the above; determining a command for the corresponding detected gesture or finger action; and transmitting the command to the image display device.

[0491] Also, in one or more embodiments, any feature (eg, function, item, step, etc.) in the method 500 may be performed by the processing unit 204 of the apparatus 200 and / or the processing unit 130 of the image display device 101.

[0492] (Special Processing System)

[0493] In some embodiments, the method 500 described herein may be performed by the processing unit 204 executing an application or by the application. The application may contain a set of instructions. In one implementation, a specialized processing system may be provided having a non-transitory medium that stores a set of instructions for the application. Execution of the instructions by the processing unit 204 of the device 200 will cause the processing unit 204 to perform the features described herein. As shown in FIG. 21, in some embodiments, the specialized processing system may include one or more servers 600. The non-transitory medium that stores instructions 610 for the application may be implemented in the server 600 configured to provide the set of instructions 610 for the device 200 to download over a network, such as over the Internet. The server 600 may be configured to obtain a request to download the set of instructions 610 from the device 200, process the request, and provide the set of instructions 610 based on the request and / or the processing of the request.

[0494] In some embodiments, processing of the request by the server 600 may include steps of verifying the device 200, verifying the user 50 of the device 200, determining the brand of the device 200, determining the model of the device 200, or any combination of the above.

[0495] It should be noted that server 600 is a specialized processing system in that it contains instructions for execution by a processing unit of a device to provide unique tangible effects in the real world. The features provided by server 600 provide improvements in the technology of image display devices and systems, as described herein.

[0496] As non-limiting examples, the instructions 610 provided by the server 600 may include: assigning a first area 206 of the screen 202 of the device 200 to sense finger actions; assigning a second area 240 of the screen 202 of the device 200 as a first button; assigning a third area 250 of the screen 202 as a second button; assigning a fourth area 260 of the screen 202 as a third button; assigning another area 270 of the screen 202 as a keyboard activation button; generating an electronic signal to cause a change in content displayed by the visual display device based on a user's finger action sensed by the assigned first area of ​​the screen of the device; ignoring input generated by a user using a portion of the touch-sensitive region that is not part of the assigned first area and that is not part of the assigned button; generating a control signal to operate a feedback component in response to the user's finger action; operating the feedback component in response to the user's finger reaching a predefined distance from a reference location; activating the feedback component and generating different control signals to generate different respective types of feedback based on different respective spatial relationships between the fingers; activating the feedback component and generating a first type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance of a first boundary; activating the feedback component and generating a second type of feedback when one or more fingers of the user cross, reach, or move to a location within a predefined distance of a second boundary; activating the feedback component based on a swipe direction; generating electronic signals to vary a size of content displayed by the image display device in response to an input signal associated with a pinch or unpinch action; obtaining an orientation of the device from an orientation sensor; generating electronic signals to vary a content displayed by the image display device based on the input signal associated with the pinch or unpinch action and the orientation of the device;generating electronic signals for moving content displayed by the image display device in response to a sensed swipe action; generating electronic signals for changing content displayed by the image display device based on an input signal associated with the swipe action and an orientation of the device; generating electronic signals for moving content if a swipe action in a first plane is sensed by the assigned first area while the device is in a first orientation; generating electronic signals for moving content if a swipe action in a second plane is sensed by the assigned first area while the device is in a second orientation different from the first orientation (the second plane being different from the first plane); generating electronic signals for moving content displayed by the image display device closer to or farther away from a user when a swipe action is sensed by the assigned first area while an orientation of the device is substantially parallel to a horizontal plane; generating electronic signals for moving content displayed by the image display device closer to or farther away from a user when a swipe action is sensed by the assigned first area while an orientation of the device is substantially perpendicular to a horizontal plane. when notified, generate electronic signals for moving content displayed by the image display device in a vertical plane within the three dimensional environment; generate electronic signals for changing content displayed by the image display device based on sensor input indicative of a sensed orientation of the device; generate electronic signals for changing the content by expanding the content in one or more directions based on sensor input indicative of the sensed orientation of the device; generate electronic signals for changing the content by rotating the content based on sensor input indicative of the sensed orientation of the device; generate electronic signals for changing the content by moving the content based on sensor input indicative of the sensed orientation of the device; obtain sensor input indicative of a sensed movement of the device; generate electronic signals for changing the content displayed by the image display device based on sensor input indicative of the sensed movement of the device; generate electronic signals for changing the content by moving the content based on sensor input indicative of the sensed movement of the device;The assigned first area may include instructions for turning off pixels of the assigned first area of ​​the screen such that the object is not displayed while the assigned first area of ​​the screen is sensing a user's finger action; operating a screen of the device to display a grid of dots within the assigned first area of ​​the screen; changing a characteristic of one or more of the dots in response to a user touching a portion of the assigned first area of ​​the screen within which one or more of the dots are displayed; or any combination of any of the foregoing.

[0497] The instructions 610 provided by the server 600 may also include instructions for detecting a gesture or finger action, such as a one finger touch movement, a two finger touch movement, a one finger swipe, a two finger swipe, a two finger pinch, a two finger unpinch, a two finger radial movement, a tap, a double tap, or any combination of the above; determining a command for the corresponding detected gesture or finger action; and transmitting the command to a visual display device.

[0498] In other embodiments, device 200 may also be considered a specialized processing system. In particular, device 200 is a specialized processing system in that it contains instructions stored in its non-transitory medium for execution by processing unit 204 to provide unique tangible effects in the real world. The features provided by device 200 (as a result of processing unit 204 executing the instructions) provide improvements in the art of image display devices and systems, as described herein.

[0499] 22 is a block diagram illustrating an embodiment of a specialized processing system 1600 that may be used to implement various features described herein. For example, in some embodiments, the processing system 1600 may be used to implement one or more of the servers 600. In other embodiments, the processing system 1600 may be used to implement the device 200. The processing system 1600 includes a bus 1602 or other communication mechanism for communicating information and a processor 1604 coupled with the bus 1602 for processing information. The processor system 1600 also includes a main memory 1606, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1602 for storing information and instructions to be executed by the processor 1604. The main memory 1606 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 1604. The processor system 1600 further includes a read only memory (ROM) 1608 or other static storage device coupled to the bus 1602 for storing static information and instructions for the processor 1604. A data storage device 1610, such as a magnetic disk, solid state disk, or optical disk, is provided and coupled to the bus 1602 for storing information and instructions.

[0500] The processor system 1600 may be coupled to a display 1612, such as a screen, via the bus 1602 to display information to a user. In some cases, if the processing system 1600 is part of a device that includes a touch screen, the display 1612 may be a touch screen. An input device 1614, including alphanumeric and other keys, is coupled to the bus 1602 to communicate information and command selections to the processor 1604. Another type of user input device is a cursor control 1616, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 1604 and for controlling cursor movement on the display 1612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allow the device to define a position in a plane. In some cases, if the processing system 1600 is part of a device that includes a touch screen, the input device 1614 and cursor control may be a touch screen.

[0501] In some embodiments, the processor system 1600 can be used to perform various functions described herein. According to some embodiments, such use is provided by the processor system 1600 in response to the processor 1604 executing one or more sequences of one or more instructions contained in the main memory 1606. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read into the main memory 1606 from another processor-readable medium, such as the storage device 1610. Execution of the sequences of instructions contained in the main memory 1606 causes the processor 1604 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in the main memory 1606. In alternative embodiments, hard-wired circuitry may be used in place of, or in combination with, software instructions to implement various embodiments described herein. Thus, the embodiments are not limited to any specific combination of hardware circuitry and software.

[0502] The term "processor-readable medium," as used herein, refers to any medium that participates in providing instructions to the processor 1604 for execution. Such media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical, solid-state, or magnetic disks, such as the storage devices 1610. Non-volatile media may be considered an example of non-transitory media. Volatile media includes dynamic memory, such as the main memory 1606. Volatile media may be considered an example of non-transitory media. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise the bus 1602. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0503] Common forms of processor-readable media include, for example, a floppy disk, a hard disk, a magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, any other physical medium with a pattern of holes, RAM, PROM, and EPROM, FLASH-EPROM, a solid-state disk, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a processor can read.

[0504] Various forms of processor-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 1604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions over a network, such as the Internet. The processing system 1600 may receive data over a network line. The bus 1602 carries the data to the main memory 1606, from which the processor 1604 reads and executes the instructions. The instructions received by the main memory 1606 may optionally be stored on a storage device 1610 either before or after execution by the processor 1604.

[0505] The processing system 1600 also includes a communication interface 1618 coupled to the bus 1602. The communication interface 1618 provides a two-way data communication coupling to a network link 1620 that is connected to a local network 1622. For example, the communication interface 1618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link may also be implemented. In any such implementation, the communication interface 1618 sends and receives electrical, electromagnetic, or optical signals that carry data streams representing various types of information.

[0506] The network link 1620 typically provides data communication through one or more networks to other devices. For example, the network link 1620 may provide a connection through a local network 1622 to a host computer 1624 or a device 1626. Data streams transported via the network link 1620 may include electrical, electromagnetic, or optical signals. The signals through the various networks and the signals on the network link 1620 and through the communication interface 1618, which carry the data to and from the processing system 1600, are exemplary forms of carrier waves transporting information. The processing system 1600 can send messages and receive data, including program code, through the networks, the network link 1620, and the communication interface 1618.

[0507] It should be noted that the apparatus 200 described herein is not limited to having the described functionality and features, and the apparatus 200 may be configured to provide other features for use with the image display device 101. Also, as noted above, in some embodiments, some or all of such functionality and features may be provided, at least in part, using an application (also referred to herein as an "app") running on the apparatus 200. As a non-limiting example, the apparatus 200 may be configured to allow a user 50 to sign up for an account associated with use of the image display device 101, log into an application using credentials configured on the account, retrieve or reset credentials for the account, switch user profiles, automatically synchronize user account data between devices, manage an account associated with use of the image display device 101, manage the configuration and use of the image display device 101, obtain supplemental learning and support functionality for the image display device 101, obtain help for troubleshooting the image display device 101 and / or the apparatus 200, access privacy, security, and data policies associated with the account, and / or access the app. access a pre-store and obtain applications for the image display device 101 and / or the device 200, perform search queries for apps, view app detail pages, view order or purchase history, receive push notifications (e.g., application updates, etc.), perform media extraction from a local device, open media files (e.g., photos, videos, etc.), select and send individual media files to the image display device 101, select and send individual intermediate files from the image display device 101 to the device 200, receive notifications regarding use of the image display device 101 (e.g., notifications from applications for the image display device 101), receive notifications regarding use of the image display device 101 from third party developers,The image display device 101 may be configured to enable managing smartphone notification settings for different apps for the image display device 101, discovering the image display device 101 or other image display devices, obtaining a battery status for the image display device 101, mirroring the image display device 101 onto the apparatus 200 (content displayed on the screen of the image display device 101 and / or the environment surrounding the user 50 as viewed through the screen and captured by the camera of the image display device 101 may be transmitted to the apparatus 200 for display by the apparatus 200), or any combination of the foregoing. The processing unit 204 of the apparatus 200 may be configured to execute instructions for providing the apparatus 200 with one or more of the above features.

[0508] Exemplary aspects of the present disclosure have been described above, together with details regarding material selection and manufacturing. As for other details of the present disclosure, these can be understood in conjunction with the aforementioned reference patents and publications, and are generally known or can be understood by those skilled in the art. The same can be true with respect to the method-based aspects of the present disclosure in terms of additional actions as generally or logically adopted.

[0509] In addition, although the present disclosure has been described with reference to several embodiments, optionally incorporating various features, the present disclosure is not limited to those described or illustrated as each variation of the disclosure is considered. Various modifications may be made to the present disclosure as described, and equivalents (whether listed herein or not included for purposes of some brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, when a range of values ​​is provided, it is to be understood that all intervening values ​​between the upper and lower limits of the range, and any other stated value or intervening values ​​within the stated range, are encompassed within the present disclosure.

[0510] It is also contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items present. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. Furthermore, it is noted that any claim may be drafted to exclude any optional element. Thus, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0511] The scope of the present disclosure should not be limited to the examples provided and / or the specification of the subject matter, but rather should be limited only by the scope of the language of the claims associated with the present disclosure. In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are therefore to be regarded in an illustrative and not a restrictive sense.

Claims

1. A touch-sensitive display; a feedback component configured to provide non-visual feedback; One or more processors An apparatus comprising: The one or more processors are operatively coupled to the touch sensitive display and the feedback component and are communicatively coupled to a wearable device, the one or more processors: receiving data from the wearable device indicating whether the device is within a field of view of a camera of the wearable device; switching between a first mode and a second mode based on the data; configured to run A device, wherein in the first mode, the one or more processors are configured to communicate one or more portions of a user interface to a user via the feedback component of the device, and in the second mode, the one or more processors are configured to present the user interface on the touch-sensitive display of the device, and the one or more processors are configured to switch to the first mode when the data indicates that the device is outside the field of view of the camera of the wearable device, and to switch to the second mode when the data indicates that the device is within the field of view of the camera of the wearable device.

2. The apparatus of claim 1 , wherein the feedback component is a tactile actuator.

3. The apparatus of claim 1 , wherein the feedback component is a speaker.

4. The apparatus of claim 1 , wherein the user interface is a graphical user interface for controlling one or more functions of the wearable device.

5. The apparatus of claim 4 , wherein the graphical user interface comprises a plurality of graphical control elements.

6. 6. The device of claim 5, wherein when in the first mode, the one or more processors are configured to refrain from presenting one or more of the plurality of graphical control elements on the touch-sensitive display.

7. The device of claim 5 , wherein when in the first mode, the one or more processors are configured to present a limited version of the graphical user interface on the touch-sensitive display.

8. 8. The device of claim 7, wherein the limited version of the graphical user interface on the touch-sensitive display is one in which one or more of the plurality of graphical control elements are not shown, one or more of the plurality of graphical control elements have a reduced brightness level, or a combination thereof.

9. 6. The device of claim 5, wherein when in the first mode, the one or more processors are configured to communicate one or more locations on the touch-sensitive display where one or more graphical control elements of the plurality of graphical control elements would otherwise be presented in the second mode.

10. 10. The apparatus of claim 9, wherein the one or more locations on the touch-sensitive display communicated in the first mode correspond to outer boundaries of one or more of the plurality of graphical control elements otherwise presented in the second mode.

11. The apparatus of claim 5 , wherein the graphical control elements correspond to a plurality of physical user input components of a dedicated input controller associated with the wearable device.

12. The device of claim 1 , wherein the one or more processors are configured to allocate areas of the touch-sensitive display for sensing different finger actions of the user when the user's finger touches the areas.

13. 13. The apparatus of claim 12, wherein the apparatus is configured to monitor a peripheral portion of the assigned area for providing a control signal to operate the feedback component, and the peripheral portion of the assigned area monitored for providing the control signal extends continuously across at least a majority of one side of the touch-sensitive display.

14. The apparatus of claim 1 , wherein the data comprises camera data.

15. 1. A processor implementation method, comprising: receiving data from a wearable device indicating whether a device is within a field of view of a camera of the wearable device; switching between a first mode and a second mode based on the data; Including, A processor-implemented method wherein in the first mode, one or more portions of a user interface are communicated to a user of the wearable device via a feedback component of the device, and in the second mode, the user interface is presented on a touch-sensitive display of the device, the feedback component of the device configured to provide non-visual feedback, and the switching includes switching to the first mode when the data indicates that the device is outside the field of view of the camera of the wearable device, and switching to the second mode when the data indicates that the device is within the field of view of the camera of the wearable device.

16. The method of claim 15 , wherein the data includes camera data.

17. 16. The method of claim 15, further comprising: allocating areas of the touch-sensitive display for sensing different finger actions of the user when the user's finger touches the areas.

18. 20. The method of claim 17, further comprising monitoring a peripheral portion of the assigned area for providing a control signal to operate the feedback component, the peripheral portion of the assigned area being monitored for providing the control signal extending continuously across at least a majority of one side of the touch-sensitive display.

Images