Method and apparatus for providing input for head-mounted image display device

Abstract

An apparatus for use with an image display device configured to be worn by a user, comprising: a screen; and a processing unit configured to specify a first area in the screen to sense a finger action of the user, where the processing unit is configured to generate an electronic signal to cause a change in content displayed by the image display apparatus based on the finger action of the user sensed by the specified first area of the screen of the apparatus.

Description

[0001] This application is a divisional application of the patent application with international application number PCT / US2019 / 038462, international application date June 21, 2019, Chinese national application number 201980054294.7, and title “Method and apparatus for providing input for a head-mounted image display device”. Technical Field

[0002] The present disclosure relates to a head-mounted image display device, and a method and apparatus for providing input to such a head-mounted image display device. Background Art

[0003] Modern computing and display technologies have facilitated the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images, or portions thereof, are presented to a user in a manner that they appear to be real or can be perceived as real. VR scenarios typically involve presenting digital or virtual image information in a manner that is opaque to actual real-world visual input. AR scenarios typically involve presenting digital or virtual image information as an enhancement to the visualization of the real world around the user (i.e., transparency to real-world visual input). Thus, AR scenarios involve presenting digital or virtual image information in a manner that is transparent to real-world visual input.

[0004] MR systems can generate and display color data, which increases the realism of the MR scene. Many of these MR systems display color data by sequentially projecting sub-images of different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Projecting the color sub-images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) can deliver a smooth color MR scene in the user's mind.

[0005] Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenes. Some such optical systems are described in U.S. Utility Patent Application No. 14 / 555,585 (Attorney Docket No. ML.20011.00), filed on November 27, 2014, the contents of which are expressly and completely incorporated herein by reference in their entirety, as if fully set forth.

[0006] MR systems can employ wearable display devices (e.g., head-mounted displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head and thereby move when the user's head moves. If the display system detects movement of the user's head, the data being displayed can be updated (e.g., "warped") to account for changes in head pose (i.e., the orientation and / or position of the user's head).

[0007] As an example, if a user wearing a head mounted display device views a virtual representation of a virtual object on a display and walks around the area where the virtual object appears, the virtual object may be rendered for each viewing point, giving the user the perception that they are walking around the object occupying real space. If the head mounted display device is used to present multiple virtual objects, measurements of head pose may be used to render the scene to match the user's dynamically changing head pose and provide increased immersion.

[0008] A head-mounted display device that enables AR provides simultaneous viewing of both real objects and virtual objects. With an "optical see-through" display, a user can see through a transparent (or translucent) element in the display system to directly view light from real objects in the environment. The transparent element (often called a "combiner") superimposes the light from the display on the user's view of the real world, where the light from the display projects an image of the virtual content on a see-through view of real objects in the environment. A camera can be mounted to the head-mounted display device to capture images or video of the scene viewed by the user.

[0009] Current optical systems, such as those in MR systems, render virtual content optically. The content is "virtual" because it does not correspond to a real physical object located at a corresponding position in space. Instead, the virtual content exists only in the brain (e.g., optical center) of the user of the head-mounted display device when the user is stimulated by a beam of light directed to their eyes.

[0010] Disclosed herein are methods and apparatus for providing input for a head-mounted image display device (eg, an MR device, an AR device, a VR device, etc.). Summary of the invention

[0011] A device for use with an image display device configured to be worn by a user, comprising: a screen; and a processing unit configured to designate a first area in the screen to sense a finger action of the user; wherein the processing unit is configured to generate an electronic signal to cause a change in content displayed by the image display device based on the finger action of the user sensed by the designated first area of ​​the screen of the device. As used in this specification, the term "finger action" may include actions performed by one or more fingers, and may include actions such as sliding, pinching, un-pinching, tapping, pressing, holding, twisting, turning, etc.

[0012] Optionally, the screen has a touch-sensitive area, and wherein the designated first area is smaller than a total area of ​​the touch-sensitive area.

[0013] Optionally, the designated first area has a shape corresponding to a shape of a screen of the image display device.

[0014] Optionally, the designated first area has a size based on a brand and / or model of the device.

[0015] Optionally, the designated first area has a size based on characteristics of the apparatus and characteristics of the image display device.

[0016] Optionally, the processing unit is configured to ignore input generated by a user using a portion of the touch-sensitive area that is not part of the designated first area and is not part of the designated button.

[0017] Optionally, the processing unit is further configured to operate the feedback component in response to the user's finger movement.

[0018] Optionally, the first area has a boundary, and wherein the user's finger action includes the user's finger crossing, reaching or moving to a position within a specified distance from the boundary.

[0019] Optionally, the first area has one or more boundaries at least partially surrounding the reference position, and wherein the processing unit is configured to operate the feedback component in response to the finger of the user 50 reaching a specified distance from the reference position.

[0020] Optionally, the processing unit is configured to operate the feedback component to generate different types of feedback based on different corresponding spatial relationships between one or more fingers of the user relative to the first area.

[0021] Optionally, the different types of feedback include a first tactile pulse having a first amplitude, and a second tactile pulse having a second amplitude different from the first amplitude.

[0022] Optionally, the different types of feedback include a first number of tactile pulses, and a second number of tactile pulses different from the first number.

[0023] Optionally, the different types of feedback include a first series of tactile pulses having a first frequency, and a second series of tactile pulses having a second frequency different from the first frequency.

[0024] Optionally, the different corresponding spatial relationships include different distances between (1) one of the one or more fingers of the user, or a point between two fingers, and (2) a reference position within the specified first area.

[0025] Optionally, the reference position comprises a center of the designated first area.

[0026] Optionally, the different distances exceed a threshold.

[0027] Optionally, the different corresponding spatial relationships include one of the one or more fingers, or a point between two fingers reaching different corresponding distances from the boundary of the designated first area.

[0028] Optionally, the designated first area has a first boundary and a second boundary; wherein the different types of feedback include at least a first type of feedback and a second type of feedback; wherein the processing unit is configured to operate the feedback component to generate the first type of feedback when one or more fingers of the user crosses, reaches or moves to a position within a specified distance from the first boundary; and wherein the processing unit is configured to operate the feedback component to generate the second type of feedback when one or more fingers of the user crosses, reaches or moves to a position within a specified distance from the second boundary.

[0029] Optionally, the first boundary includes a left boundary or a right boundary of the designated first region, and the second boundary includes a top boundary or a bottom boundary of the designated first region.

[0030] Optionally, the processing unit is configured to operate the feedback component based on the sliding direction.

[0031] Optionally, the processing unit is configured to obtain an input signal associated with a contraction or expansion action performed on the designated first area of ​​the screen.

[0032] Optionally, the processing unit is configured to generate an electronic signal based on an input signal associated with the contraction or expansion action to cause a size of content displayed by the image display device to change.

[0033] Optionally, the apparatus further comprises an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is further configured to generate an electronic signal based on an input signal associated with the contraction or expansion motion and the sensed orientation of the apparatus to cause content displayed by the image display device to change.

[0034] Optionally, the processing unit is configured to generate an electronic signal to cause the content to shrink and expand in a first plane if a contraction or expansion motion is sensed by the device when the device is in a first orientation; and wherein the processing unit is configured to generate an electronic signal to cause the content to shrink and expand in a second plane if a contraction or expansion motion is sensed by the device when the device is in a second orientation different from the first orientation, and the second plane is different from the first plane.

[0035] Optionally, the device has a first orientation when a major axis of the device forms an angle of less than 45° with a horizontal plane.

[0036] Optionally, the device has a second orientation when a major axis of the device forms an angle of less than 45° with a vertical plane.

[0037] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0038] Optionally, the first plane and the second plane are relative to the virtual three-dimensional environment.

[0039] Optionally, the first plane is perpendicular to the second plane.

[0040] Optionally, the processing unit is configured to obtain an input signal associated with a sliding action performed on the designated first area of ​​the screen.

[0041] Optionally, the processing unit is configured to generate an electronic signal in response to an input signal associated with a sliding motion to cause the content to change by moving the content.

[0042] Optionally, the apparatus further comprises an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is configured to generate an electronic signal based on an input signal associated with the sliding action and the sensed orientation of the apparatus to cause content displayed by the image display device to change.

[0043] Optionally, the processing unit is configured to generate an electronic signal to move the content in a first plane if a sliding motion is sensed by the device when the device is in a first orientation; and wherein the processing unit is configured to generate an electronic signal to move the content in a second plane if a sliding motion is sensed by the device when the device is in a second orientation different from the first orientation, the second plane being different from the first plane.

[0044] Optionally, the device has a first orientation when a major axis of the device forms an angle of less than 45° with a horizontal plane.

[0045] Optionally, the device has a second orientation when a major axis of the device forms an angle of less than 45° with a vertical plane.

[0046] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0047] Optionally, the first plane and the second plane are relative to the virtual three-dimensional environment.

[0048] Optionally, the first plane is perpendicular to the second plane.

[0049] Optionally, the content is in a virtual three-dimensional environment, and wherein the processing unit is configured to generate an electronic signal when a sliding motion is sensed by the device when the device is oriented substantially parallel to a horizontal plane to cause content displayed by the image display device to change by moving the content closer to or further away from the user.

[0050] Optionally, the content is in a virtual three-dimensional environment, and wherein the processing unit is configured to generate an electronic signal to cause content displayed by the image display device to move in a vertical plane in the three-dimensional environment when a sliding motion is sensed by the device when the device is oriented substantially perpendicular to a horizontal plane.

[0051] Optionally, the device further comprises an orientation sensor for sensing an orientation of the device, wherein the processing unit is configured to generate the electronic signal based on the sensed orientation of the device.

[0052] Optionally, the processing unit is configured to generate an electronic signal based on the sensed orientation of the device to cause the content to expand in one or more directions.

[0053] Optionally, the processing unit is configured to generate an electronic signal based on the sensed orientation of the device to rotate the content.

[0054] Optionally, the processing unit is configured to generate an electronic signal based on the sensed orientation of the apparatus to cause the content to move.

[0055] Optionally, the apparatus further comprises a movement sensor for sensing movement of the apparatus, wherein the processing unit is configured to generate an electronic signal based on the sensed movement of the apparatus to cause content displayed by the image display device to change.

[0056] Optionally, the processing unit is configured to generate an electronic signal based on the sensed movement of the apparatus to cause the content to change by causing the content to move.

[0057] Optionally, the device is a handheld device.

[0058] Optionally, the handheld device comprises a cellular phone, a smart phone, a personal digital assistant (PDA), or a tablet computer.

[0059] Optionally, when the designated first area of ​​the screen is sensing the user's finger motion, the designated first area of ​​the screen has no displayed object.

[0060] Optionally, the processing unit is configured to operate the screen to display a grid of dots in the designated first area of ​​the screen.

[0061] Optionally, the processing unit is further configured to change a characteristic of one or more of the points in response to a user touching a portion of one or more of the displayed points in the designated first area of ​​the screen.

[0062] Optionally, the processing unit is further configured to designate a second area of ​​the screen as the first button.

[0063] Optionally, the first button is a "home" button.

[0064] Optionally, the processing unit is further configured to designate a third area of ​​the screen as a second button.

[0065] Optionally, the second button is a "Toggle" button.

[0066] Optionally, the processing unit is further configured to designate a fourth area of ​​the screen as a third button.

[0067] Optionally, the third button is a "buffer" button.

[0068] Optionally, the processing unit is further configured to designate a second area of ​​the screen as a keyboard activation button, and the processing unit is configured to operate the screen to display the keyboard in response to the user touching the designated second area of ​​the screen.

[0069] Optionally, the apparatus further comprises a wireless receiver for communicating with the image display device.

[0070] Optionally, the apparatus further comprises a connector for communicating with an image display device via a cable.

[0071] Optionally, the apparatus further comprises a non-transitory medium storing a set of instructions, execution of which will cause the processing unit to designate a first area of ​​the screen for sensing a finger movement of the user.

[0072] Optionally, the change in the content includes a change in the size of the content, a change in the position of the content, a change in the shape of the content, a change in the color of the content, a replacement of information in the content, an increase or decrease in the amount of information in the content, or any combination of the foregoing.

[0073] A method includes: designating a first area in a screen of an apparatus for sensing finger movements of a user of an image display device, wherein the image display device is configured to be worn on the head of the user, and wherein the apparatus is different from the image display device; generating an electronic signal based on the user's finger movements sensed by the designated first area of ​​the screen of the apparatus to cause a change in content displayed by the image display device.

[0074] Optionally, the screen has a touch-sensitive area, and wherein the designated first area is smaller than a total area of ​​the touch-sensitive area.

[0075] Optionally, the designated first area has a shape corresponding to a shape of a screen of the image display device.

[0076] Optionally, the designated first area has a size based on a brand and / or model of the device.

[0077] Optionally, the designated first area has a size based on characteristics of the apparatus and characteristics of the image display device.

[0078] Optionally, the method may further include ignoring input generated by a user using a portion of the touch-sensitive area that is not part of the designated first area and is not part of the designated button.

[0079] Optionally, the method may further include generating a control signal in response to the user's finger motion to operate the feedback component.

[0080] Optionally, the first area has a boundary, and wherein the user's finger action includes the user's finger crossing, reaching or moving to a position within a specified distance from the boundary.

[0081] Optionally, the first region has one or more boundaries at least partially surrounding the reference position, and wherein the control signal is used to operate the feedback component in response to the user's finger reaching a specified distance from the reference position.

[0082] Optionally, the method further comprises generating different control signals to operate the feedback component to generate different corresponding types of feedback based on different corresponding spatial relationships between one or more fingers of the user relative to the first area.

[0083] Optionally, the different types of feedback include a first tactile pulse having a first amplitude, and a second tactile pulse having a second amplitude different from the first amplitude.

[0084] Optionally, the different types of feedback include a first number of tactile pulses, and a second number of tactile pulses different from the first number.

[0085] Optionally, the different types of feedback include a first series of tactile pulses having a first frequency, and a second series of tactile pulses having a second frequency different from the first frequency.

[0086] Optionally, the different corresponding spatial relationships include different distances between (1) one of the one or more fingers of the user, or a point between two fingers, and (2) a reference position within the specified first area.

[0087] Optionally, the reference position comprises a center of the designated first area.

[0088] Optionally, the different distances exceed a threshold.

[0089] Optionally, the different corresponding spatial relationships include one of the one or more fingers, or a point between two fingers reaching different corresponding distances from the boundary of the designated first area.

[0090] Optionally, the designated first area has a first boundary and a second boundary; wherein the different types of feedback include at least a first type of feedback and a second type of feedback; wherein, when one or more fingers of the user cross, reach or move to a position within a specified distance from the first boundary, the feedback component is operated to generate the first type of feedback; and wherein when one or more fingers of the user cross, reach or move to a position within a specified distance from the second boundary, the feedback component is operated to generate the second type of feedback.

[0091] Optionally, the first boundary includes a left boundary or a right boundary of the designated first region, and the second boundary includes a top boundary or a bottom boundary of the designated first region.

[0092] Optionally, the control signal for operating the feedback component is based on the sliding direction.

[0093] Optionally, the method further includes receiving an input signal associated with a contraction or expansion action performed by the user on the designated first area 210 of the screen.

[0094] Optionally, the electronic signal is used to respond to an input signal associated with a contraction or expansion action to cause a size change of content displayed by the image display device.

[0095] Optionally, the method further comprises obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is used to cause content displayed by the image display device to change based on the input signal associated with the contraction or expansion motion and the orientation of the apparatus.

[0096] Optionally, when the device is in a first orientation, if a contraction or expansion motion is sensed by the designated first area, the content is changed by shrinking and expanding the content in a first plane; and wherein, when the device is in a second orientation, if a contraction or expansion motion is sensed by the designated first area, the content is changed by shrinking and expanding the content in a second plane, the second plane being different from the first plane.

[0097] Optionally, the device has a first orientation when a major axis of the device forms an angle of less than 45° with a horizontal plane.

[0098] Optionally, the device has a second orientation when a major axis of the device forms an angle of less than 45° with a vertical plane.

[0099] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0100] Optionally, the first plane and the second plane are relative to the virtual three-dimensional environment.

[0101] Optionally, the first plane is perpendicular to the second plane.

[0102] Optionally, the method further comprises receiving an input signal associated with a sliding action performed by a user on the designated first area in the screen.

[0103] Optionally, the electronic signal is used to move content displayed by the image display device in response to the sensed sliding motion.

[0104] Optionally, the method further comprises obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is used to cause content displayed by the image display device to change based on the input signal associated with the sliding action and the orientation of the apparatus.

[0105] Optionally, when the device is in a first orientation, if a sliding motion is sensed by the designated first area, the content is changed by moving the content in a first plane; and wherein, when the device is in a second orientation different from the first orientation, if a sliding motion is sensed by the designated first area, the content is changed by moving the content in a second plane, the second plane being different from the first plane.

[0106] Optionally, the device has a first orientation when a major axis of the device forms an angle of less than 45° with a horizontal plane.

[0107] Optionally, the device has a second orientation when a major axis of the device forms an angle of less than 45° with a vertical plane.

[0108] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0109] Optionally, the first plane and the second plane are relative to the virtual three-dimensional environment.

[0110] Optionally, the first plane is perpendicular to the second plane.

[0111] Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is used to cause the content displayed by the image display device to move closer to or farther away from the user when a sliding motion is sensed by the designated first area when the device is oriented substantially parallel to a horizontal plane.

[0112] Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is used to cause content displayed by the image display device to move in a vertical plane in the three-dimensional environment when a sliding motion is sensed by the designated first area when the device is oriented substantially perpendicular to a horizontal plane.

[0113] Optionally, the method further comprises obtaining a sensor input indicating a sensed orientation of the device, wherein the electronic signal is used to cause content displayed by the image display apparatus to change based on the sensor input indicating the sensed orientation of the device.

[0114] Optionally, the electronic signal is used to cause the content to change by causing the content to expand in one or more directions based on sensor input indicating a sensed orientation of the device.

[0115] Optionally, the electronic signal is used to cause the content to change by rotating the content based on sensor input indicating a sensed orientation of the device.

[0116] Optionally, the electronic signal is used to cause the content to change by causing the content to move based on sensor input of the sensed orientation of the pointing device.

[0117] Optionally, the method further comprises obtaining a sensor input of the sensed movement of the pointing device, wherein the electronic signal is used to cause content displayed by the image display device to change based on the sensor input of the sensed movement of the pointing device.

[0118] Optionally, the electronic signal is used to cause the content to change by causing the content to move based on sensor input of sensed movement of the pointing device.

[0119] Optionally, the device is a handheld device.

[0120] Optionally, the handheld device comprises a cellular phone, a smart phone, a personal digital assistant (PDA), or a tablet computer.

[0121] Optionally, when the designated first area of ​​the screen is sensing the user's finger motion, the designated first area of ​​the screen has no displayed object.

[0122] Optionally, the method further comprises operating the screen to display a grid of points in the designated first area of ​​the screen.

[0123] Optionally, the method further comprises changing a characteristic of one or more of the points in response to a user touching a portion of one or more of the displayed points in the designated first area of ​​the screen.

[0124] Optionally, the method further comprises designating a second area of ​​the screen as the first button.

[0125] Optionally, the first button is a "home" button.

[0126] Optionally, the method further comprises designating a third area of ​​the screen as a second button.

[0127] Optionally, the second button is a "Toggle" button.

[0128] Optionally, the method further comprises designating a fourth area of ​​the screen as a third button.

[0129] Optionally, the third button is a "buffer" button.

[0130] Optionally, the method further comprises: designating a second area of ​​the screen as a keyboard activation button, and in response to the user touching the designated second area of ​​the screen, operating the screen to display the keyboard.

[0131] Optionally, the method further comprises communicating wirelessly with the image display device.

[0132] Optionally, the method further comprises communicating with an image display device via a cable.

[0133] Optionally, the apparatus comprises a non-transitory medium storing instructions, and wherein the action of designating a first area of ​​the screen for sensing a finger action of the user is performed based on the instructions.

[0134] Optionally, the change in the content includes a change in the size of the content, a change in the position of the content, a change in the shape of the content, a change in the color of the content, a replacement of information in the content, an increase or decrease in the amount of information in the content, or any combination of the foregoing.

[0135] A product comprising a non-transitory medium storing a set of instructions, the execution of which will cause a method to be performed, the method comprising: designating a first area in a screen of an apparatus for sensing finger movements of a user of an image display device, wherein the image display device is configured to be worn by the user's head, and wherein the apparatus is different from the image display device; generating an electronic signal based on the user's finger movements sensed by the designated first area of ​​the screen of the apparatus to cause a change in content displayed by the image display device.

[0136] Optionally, the screen has a touch-sensitive area, and wherein the designated first area is smaller than a total area of ​​the touch-sensitive area.

[0137] Optionally, the designated first area has a shape corresponding to a shape of a screen of the image display device.

[0138] Optionally, the designated first area has a size based on a brand and / or model of the device.

[0139] Optionally, the designated first area has a size based on characteristics of the apparatus and characteristics of the image display device.

[0140] Optionally, the method may further include ignoring input generated by a user using a portion of the touch-sensitive area that is not part of the designated first area and is not part of the designated button.

[0141] Optionally, the method may further include generating a control signal in response to the user's finger motion to operate the feedback component.

[0142] Optionally, the first area has a boundary, and wherein the user's finger action includes the user's finger crossing, reaching or moving to a position within a specified distance from the boundary.

[0143] Optionally, the first region has one or more boundaries at least partially surrounding the reference position, and wherein the control signal is used to operate the feedback component in response to the user's finger reaching a specified distance from the reference position.

[0144] Optionally, the method further comprises generating different control signals to operate the feedback component to generate different corresponding types of feedback based on different corresponding spatial relationships between one or more fingers of the user relative to the first area.

[0145] Optionally, the different types of feedback include a first tactile pulse having a first amplitude, and a second tactile pulse having a second amplitude different from the first amplitude.

[0146] Optionally, the different types of feedback include a first number of tactile pulses, and a second number of tactile pulses different from the first number.

[0147] Optionally, the different types of feedback include a first series of tactile pulses having a first frequency, and a second series of tactile pulses having a second frequency different from the first frequency.

[0148] Optionally, the different corresponding spatial relationships include different distances between (1) one of the one or more fingers of the user, or a point between two fingers, and (2) a reference position within the specified first area.

[0149] Optionally, the reference position comprises a center of the designated first area.

[0150] Optionally, the different distances exceed a threshold.

[0151] Optionally, the different corresponding spatial relationships include one of the one or more fingers, or a point between two fingers reaching different corresponding distances from the boundary of the designated first area.

[0152] Optionally, the designated first area has a first boundary and a second boundary; wherein the different types of feedback include at least a first type of feedback and a second type of feedback; wherein, when one or more fingers of the user cross, reach or move to a position within a specified distance from the first boundary, the feedback component is operated to generate the first type of feedback; and wherein when one or more fingers of the user cross, reach or move to a position within a specified distance from the second boundary, the feedback component is operated to generate the second type of feedback.

[0153] Optionally, the first boundary includes a left boundary or a right boundary of the designated first region, and the second boundary includes a top boundary or a bottom boundary of the designated first region.

[0154] Optionally, the control signal for operating the feedback component is based on the sliding direction.

[0155] Optionally, the method further includes receiving an input signal associated with a contraction or expansion action performed by the user on the designated first area 210 of the screen.

[0156] Optionally, the electronic signal is used to respond to an input signal associated with a contraction or expansion action to cause a size change of content displayed by the image display device.

[0157] Optionally, the method further comprises obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is used to cause content displayed by the image display device to change based on the input signal associated with the contraction or expansion motion and the orientation of the apparatus.

[0158] Optionally, when the device is in a first orientation, if a contraction or expansion motion is sensed by the designated first area, the content is changed by shrinking and expanding the content in a first plane; and wherein, when the device is in a second orientation, if a contraction or expansion motion is sensed by the designated first area, the content is changed by shrinking and expanding the content in a second plane, the second plane being different from the first plane.

[0159] Optionally, the device has a first orientation when a major axis of the device forms an angle of less than 45° with a horizontal plane.

[0160] Optionally, the device has a second orientation when a major axis of the device forms an angle of less than 45° with a vertical plane.

[0161] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0162] Optionally, the first plane and the second plane are relative to the virtual three-dimensional environment.

[0163] Optionally, the first plane is perpendicular to the second plane.

[0164] Optionally, the method further comprises receiving an input signal associated with a sliding action performed by a user on the designated first area in the screen.

[0165] Optionally, the electronic signal is used to move content displayed by the image display device in response to the sensed sliding motion.

[0166] Optionally, the method further comprises obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is used to cause content displayed by the image display device to change based on the input signal associated with the sliding action and the orientation of the apparatus.

[0167] Optionally, when the device is in a first orientation, if a sliding motion is sensed by the designated first area, the content is changed by moving the content in a first plane; and wherein, when the device is in a second orientation different from the first orientation, if a sliding motion is sensed by the designated first area, the content is changed by moving the content in a second plane, the second plane being different from the first plane.

[0168] Optionally, the device has a first orientation when a major axis of the device forms an angle of less than 45° with a horizontal plane.

[0169] Optionally, the device has a second orientation when a major axis of the device forms an angle of less than 45° with a vertical plane.

[0170] Optionally, the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0171] Optionally, the first plane and the second plane are relative to the virtual three-dimensional environment.

[0172] Optionally, the first plane is perpendicular to the second plane.

[0173] Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is used to cause the content displayed by the image display device to move closer to or farther away from the user when a sliding motion is sensed by the designated first area when the device is oriented substantially parallel to a horizontal plane.

[0174] Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is used to cause content displayed by the image display device to move in a vertical plane in the three-dimensional environment when a sliding motion is sensed by the designated first area when the device is oriented substantially perpendicular to a horizontal plane.

[0175] Optionally, the method further comprises obtaining a sensor input indicating a sensed orientation of the device, wherein the electronic signal is used to cause content displayed by the image display apparatus to change based on the sensor input indicating the sensed orientation of the device.

[0176] Optionally, the electronic signal is used to cause the content to change by causing the content to expand in one or more directions based on sensor input indicating a sensed orientation of the device.

[0177] Optionally, the electronic signal is used to cause the content to change by rotating the content based on sensor input indicating a sensed orientation of the device.

[0178] Optionally, the electronic signal is used to cause the content to change by causing the content to move based on sensor input of the sensed orientation of the pointing device.

[0179] Optionally, the method further comprises obtaining a sensor input of the sensed movement of the pointing device, wherein the electronic signal is used to cause content displayed by the image display device to change based on the sensor input of the sensed movement of the pointing device.

[0180] Optionally, the electronic signal is used to cause the content to change by causing the content to move based on sensor input of sensed movement of the pointing device.

[0181] Optionally, the device is a handheld device.

[0182] Optionally, the handheld device comprises a cellular phone, a smart phone, a personal digital assistant (PDA), or a tablet computer.

[0183] Optionally, when the designated first area of ​​the screen is sensing the user's finger motion, the designated first area of ​​the screen has no displayed object.

[0184] Optionally, the method further comprises operating the screen to display a grid of points in the designated first area of ​​the screen.

[0185] Optionally, the method further comprises changing a characteristic of one or more of the points in response to a user touching a portion of one or more of the displayed points in the designated first area of ​​the screen.

[0186] Optionally, the method further comprises designating a second area of ​​the screen as the first button.

[0187] Optionally, the first button is a "home" button.

[0188] Optionally, the method further comprises designating a third area of ​​the screen as a second button.

[0189] Optionally, the second button is a "Toggle" button.

[0190] Optionally, the method further comprises designating a fourth area of ​​the screen as a third button.

[0191] Optionally, the third button is a "buffer" button.

[0192] Optionally, the method further comprises: designating a second area of ​​the screen as a keyboard activation button, and in response to the user touching the designated second area of ​​the screen, operating the screen to display the keyboard.

[0193] Optionally, the method further comprises communicating wirelessly with the image display device.

[0194] Optionally, the method further comprises communicating with an image display device via a cable.

[0195] Optionally, the product includes instructions for designating a first area of ​​the screen for an action of sensing a user's finger motion.

[0196] Optionally, the change in the content includes a change in the size of the content, a change in the position of the content, a change in the shape of the content, a change in the color of the content, a replacement of information in the content, an increase or decrease in the amount of information in the content, or any combination of the foregoing.

[0197] A computing device includes: a proximity-sensitive display; and one or more processors operably coupled to the proximity-sensitive display and communicatively coupled to a wearable computing system, the one or more processors being configured to: receive data from the wearable computing system indicating whether the proximity-sensitive display is visible to a user of the wearable computing system; monitor the data received from the wearable computing system for changes in visibility of the proximity-sensitive display to the user; and in response to detecting a change in visibility of the proximity-sensitive display to the user, switch between: (i) a first presentation mode in which the one or more processors are configured to operate the proximity-sensitive display such that a first load is placed on a power supply of the computing device, and (ii) a second presentation mode in which the one or more processors are configured to operate the proximity-sensitive display such that a second load is placed on the power supply of the computing device, the second load being less than the first load.

[0198] Optionally, the first presentation mode is a mode in which the one or more processors are configured to present a graphical user interface on the proximity-sensitive display.

[0199] Optionally, the second presentation mode is a mode in which the one or more processors are configured to present a limited version of the graphical user interface on the proximity-sensitive display.

[0200] Optionally, the first presentation mode is a mode in which one or more processors are configured to operate the proximity-sensitive display so that content is presented at a first brightness level, and the second presentation mode is a mode in which one or more processors are configured to operate the proximity-sensitive display so that content is presented at a second brightness level that is lower than the first brightness level.

[0201] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0202] Optionally, the computing device is configured to perform one or more of the operations described in one or more of the methods described immediately below.

[0203] A computer-implemented method includes: receiving a first set of data from a wearable computing system, the first set of data indicating whether a proximity-sensitive display is visible to a user of the wearable computing system at a first point in time; receiving a second set of data from the wearable computing system, the second set of data indicating whether the proximity-sensitive display is visible to the user of the wearable computing system at a second, later point in time; determining that a change in visibility of the proximity-sensitive display to the user has occurred based on receiving the first set of data and the second set of data from the wearable computing system; and in response to determining that a change in visibility of the proximity-sensitive display to the user has occurred, switching between: (i) a first presentation mode in which a graphical user interface is presented on the proximity-sensitive display, and (ii) a second presentation mode in which the proximity-sensitive display consumes less power than in the first presentation mode.

[0204] Optionally, based on receiving the first set of data and the second set of data from the wearable computing system, determining a change in user visibility of the proximity-sensitive display includes: based on receiving the first set of data and the second set of data from the wearable computing system, determining that the user has lost visibility of the proximity-sensitive display.

[0205] Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method includes switching from the first presentation mode to the second presentation mode.

[0206] Optionally, determining a change in user visibility of the proximity-sensitive display based on receiving the first set of data and the second set of data from the wearable computing system includes: determining that the user has regained visibility of the proximity-sensitive display based on receiving the first set of data and the second set of data from the wearable computing system.

[0207] Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method includes switching from the second presentation mode to the first presentation mode.

[0208] Optionally, the method may be performed by the computing device described immediately above and / or the computing device described immediately below.

[0209] A computing device includes: a proximity-sensitive display; a feedback component; and one or more processors operably coupled to the proximity-sensitive display and the feedback component and communicatively coupled to a wearable computing system, the one or more processors being configured to: receive data from the wearable computing system indicating whether the proximity-sensitive display is visible to a user of the wearable computing system; monitor the data received from the wearable computing system for changes in visibility of the proximity-sensitive display to the user; and in response to detecting a change in visibility of the proximity-sensitive display to the user, switch between: (i) a first mode in which the one or more processors are configured to present a user interface on the proximity-sensitive display, and (ii) a second mode in which the one or more processors are configured to communicate one or more portions of the user interface to the user by using the feedback component.

[0210] Optionally, the feedback component is a tactile actuator.

[0211] Optionally, the feedback component is a speaker.

[0212] Optionally, the user interface is a graphical user interface for controlling one or more functions of the wearable computing system.

[0213] Optionally, the graphical user interface includes a plurality of graphical control elements.

[0214] Optionally, the second mode is a mode in which the one or more processors are configured to suppress presentation of one or more of the plurality of graphical control elements on the proximity-sensitive display.

[0215] Optionally, the second mode is a mode in which the one or more processors are configured to present a limited version of the graphical user interface on the proximity-sensitive display.

[0216] Optionally, the limited version of the graphical user interface on the proximity-sensitive display is a version in which one or more of the plurality of graphical control elements are not shown, a version in which a brightness level of one or more of the plurality of graphical control elements is reduced, or a combination thereof.

[0217] Optionally, the second mode is a mode in which the one or more processors are configured to communicate one or more locations on the proximity-sensitive display at which one or more of the plurality of graphical control elements were presented in the first mode.

[0218] Optionally, the one or more locations on the proximity-sensitive display communicated in the second mode correspond to one or more outer boundaries of the plurality of graphical control elements presented in the first mode.

[0219] Optionally, the plurality of graphical control elements correspond to a plurality of physical user input components in a dedicated input controller associated with the wearable computing system.

[0220] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0221] Optionally, the computing device is configured to perform one or more of the operations in the method described immediately above and / or in the method described immediately below.

[0222] A computer-implemented method includes: receiving a first set of data from a wearable computing system, the first set of data indicating whether a proximity-sensitive display is visible to a user of the wearable computing system at a first point in time; receiving a second set of data from the wearable computing system, the second set of data indicating whether the proximity-sensitive display is visible to the user of the wearable computing system at a second, later point in time; determining that a change in visibility of the proximity-sensitive display to the user has occurred based on receiving the first set of data and the second set of data from the wearable computing system; and in response to determining that a change in visibility of the proximity-sensitive display to the user has occurred, switching between: (i) a first mode in which a user interface is presented on the proximity-sensitive display, and (ii) a second mode in which one or more portions of the user interface are communicated to the user of the wearable computing system by using a feedback component.

[0223] Optionally, based on receiving the first set of data and the second set of data from the wearable computing system, determining a change in user visibility of the proximity-sensitive display includes: based on receiving the first set of data and the second set of data from the wearable computing system, determining that the user has lost visibility of the proximity-sensitive display.

[0224] Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method comprises switching from the first mode to the second mode.

[0225] Optionally, determining a change in user visibility of the proximity-sensitive display based on receiving the first set of data and the second set of data from the wearable computing system includes: determining that the user has regained visibility of the proximity-sensitive display based on receiving the first set of data and the second set of data from the wearable computing system.

[0226] Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method comprises switching from the second mode to the first mode.

[0227] Optionally, the method may be performed by one or more of the computing devices described immediately above.

[0228] A wearable computing system includes: a head-mounted display configured to be worn on a user's head; one or more sensing devices configured to monitor an environment of the head-mounted display; and one or more processors operably coupled to the head-mounted display and the one or more sensing devices and communicatively coupled to a computing device, the one or more processors configured to: determine whether a display of the computing device is visible to a user of the wearable computing system using data obtained from the one or more sensing devices; generate one or more messages based at least in part on whether the display of the computing device is determined to be visible to the user of the wearable computing system; and send the one or more messages to the computing device.

[0229] Optionally, the one or more sensing devices include one or more cameras.

[0230] Optionally, the one or more cameras include one or more forward-facing cameras configured to capture images of an environment of a user of the wearable computing system.

[0231] Optionally, the one or more cameras include one or more inward-facing cameras configured to capture images of one or more of the user's eyes, one or more portions of the user's head or face, or a combination thereof.

[0232] Optionally, the one or more sensing devices include one or more proximity sensors.

[0233] Optionally, the wearable computing system is configured to operate in the same or similar manner as one or more of the wearable computing systems, wearable display systems, and / or image display devices described herein.

[0234] Optionally, the wearable computing system is configured to perform one or more operations in the method described immediately below.

[0235] A computer-implemented method includes obtaining data from one or more sensing devices of a wearable computing system; determining whether a display of a computing device is visible to a user of the wearable computing system based on receiving the data from the one or more sensing devices of the wearable computing system; generating one or more messages based on whether the display of the computing device is determined to be visible to the user of the wearable computing system; and sending the one or more messages to the computing device.

[0236] Optionally, obtaining data from one or more sensing devices of the wearable computing system includes obtaining data from one or more forward-facing cameras configured to capture images of an environment of a user of the wearable computing system.

[0237] Optionally, based on receiving data from one or more sensing devices of the wearable computing system, determining whether a display of the computing device is visible to a user of the wearable computing system includes determining whether the display of the computing device is shown in one or more images captured by one or more forward-facing cameras.

[0238] Optionally, obtaining data from one or more sensing devices of the wearable computing system includes obtaining data from one or more inward-facing cameras configured to capture images of one or more of the user's eyes, one or more portions of the user's head or face, or a combination thereof.

[0239] Optionally, based on receiving data from one or more sensing devices of the wearable computing system, determining whether a display of the computing device is visible to a user of the wearable computing system includes: determining whether the wearable computing system is being worn by the user on the user's head based on one or more images captured by one or more inward-facing cameras.

[0240] Optionally, based on one or more images captured by the one or more inward-facing cameras, determining whether the wearable computing system is being worn by the user on the user's head includes determining whether the user is shown in the one or more images captured by the one or more inward-facing cameras.

[0241] Optionally, obtaining data from one or more sensing devices of the wearable computing system includes obtaining data from one or more proximity sensors.

[0242] Optionally, based on receiving data from one or more sensing devices of the wearable computing system, determining whether a display of the computing device is visible to a user of the wearable computing system includes determining whether data received from one or more proximity sensors indicates that the wearable computing system is being worn by the user on the user's head.

[0243] Optionally, the method may be performed by the wearable computing system described immediately above.

[0244] A computing device includes: a proximity-sensitive display; a feedback component; and one or more processors operably coupled to the proximity-sensitive display and the feedback component and communicatively coupled to a wearable computing system, the one or more processors being configured to: receive data indicating a first touch input received at a first location on the proximity-sensitive display; receive data indicating a second touch input received at a second location on the proximity-sensitive display immediately following the first touch input; determine whether each of the first location and the second location falls within a specific area of ​​the proximity-sensitive display; in response to determining that one or both of the first location and the second location fall within the specific area of ​​the proximity-sensitive display: generate one or more messages based at least in part on one or both of the first touch input and the second touch input; and send the one or more messages generated based at least in part on one or both of the first touch input and the second touch input to the wearable computing system; and in response to (i) determining that the first location falls within the specific area of ​​the proximity-sensitive display, and (ii) determining that the second location falls outside the specific area of ​​the proximity-sensitive display: cause the feedback component to generate feedback.

[0245] Optionally, specific areas of the proximity-sensitive display are associated with specific control elements.

[0246] Optionally, the specific control element is a graphical control element, and the one or more processors are further configured to display the specific control element at a location on the proximity-sensitive display that coincides with the specific area.

[0247] Optionally, the feedback component is a tactile actuator.

[0248] Optionally, the feedback component is a speaker.

[0249] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0250] Optionally, the computing device is configured to perform one or more operations in the method described immediately below.

[0251] A computer-implemented method includes: receiving data indicating a first touch input received at a first location on a proximity-sensitive display; receiving data indicating a second touch input received at a second location on the proximity-sensitive display immediately following the first touch input; determining whether each of the first location and the second location falls within a specific area of ​​the proximity-sensitive display; in response to determining that one or both of the first location and the second location fall within the specific area of ​​the proximity-sensitive display: generating one or more messages based at least in part on one or both of the first touch input and the second touch input; and sending the one or more messages generated based at least in part on one or both of the first touch input and the second touch input to a wearable computing system; and in response to (i) determining that the first location falls within the specific area of ​​the proximity-sensitive display, and (ii) determining that the second location falls outside the specific area of ​​the proximity-sensitive display: providing feedback for output via a feedback component.

[0252] Optionally, the method may be performed by the computing device described immediately above.

[0253] A computing device includes: a proximity-sensitive display; an orientation sensor configured to sense the orientation of the computing device; and one or more processors operably coupled to the proximity-sensitive display and the orientation sensor and communicatively coupled to a wearable computing system, the one or more processors configured to: monitor a touch input sequence corresponding to any one of a plurality of different predefined gestures received through the proximity-sensitive display; in response to detecting any of the plurality of different predefined gestures: respectively select a particular type of transformation associated with the detected gesture from a plurality of different types of transformations associated with the plurality of different predefined gestures; select a particular axis from a plurality of different axes based on data obtained from the orientation sensor; generate a command to apply the particular type of transformation to a virtual object relative to the particular axis; and send the command to the wearable computing system. Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatuses described herein.

[0254] Optionally, the computing device is configured to perform one or more operations in the method described immediately below.

[0255] A computer-implemented method includes: obtaining data indicating an orientation of a computing device; receiving data indicating a sequence of touch inputs received through a proximity-sensitive display of the computing device; determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to a particular gesture; and in response to determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to the particular gesture: selecting a particular type of transformation associated with the particular gesture from a plurality of different types of transformations associated with a plurality of different predefined gestures; and selecting a particular axis from a plurality of different axes based on the orientation of the device; and generating a command to apply the particular type of transformation to a virtual object relative to the particular axis.

[0256] Optionally, the plurality of different types of transformations include one or more of: rotation, translation, and resizing.

[0257] Optionally, the method may be performed by the computing device described immediately above.

[0258] A computing device includes: a proximity-sensitive display; and one or more processors operably coupled to the proximity-sensitive display and communicatively coupled to a wearable display system, the one or more processors being configured to: present a specific content segment on the proximity-sensitive display; during presentation of the specific content segment on the proximity-sensitive display, monitor a touch input sequence corresponding to any one of a plurality of different predefined gestures received through the proximity-sensitive display; in response to detecting any one of a plurality of different predefined gestures during presentation of the specific content segment on the proximity-sensitive display: generate one or more messages indicating that presentation of the specific content segment will be switched to the wearable display system; send the one or more messages to the wearable display system; and stop presenting the specific content segment on the proximity-sensitive display.

[0259] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0260] Optionally, the computing device is configured to perform one or more operations in the method described immediately below.

[0261] A computer-implemented method includes: presenting a particular content segment on a proximity-sensitive display; receiving data indicating a sequence of touch inputs received through the proximity-sensitive display while the particular content segment is presented on the proximity-sensitive display; determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to a particular gesture; in response to determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to the particular gesture: generating one or more messages indicating that presentation of the particular content segment will be switched to a wearable display system; sending the one or more messages to the wearable display system; and ceasing presentation of the particular content segment on the proximity-sensitive display.

[0262] Optionally, determining that a sequence of touch inputs received through a proximity-sensitive display corresponds to a particular gesture includes determining that a sequence of touch inputs received through a proximity-sensitive display corresponds to a gesture in which a user has dragged one or more of their fingers from a location on the proximity-sensitive display associated with a particular content segment to an edge of the proximity-sensitive display.

[0263] Optionally, determining that a sequence of touch inputs received through a proximity-sensitive display corresponds to a particular gesture includes determining that a sequence of touch inputs received through a proximity-sensitive display corresponds to a gesture in which a user pinches two or more of his or her fingers together at a particular location on the proximity-sensitive display associated with a particular content segment, and then moves the two or more pinched fingers away from the particular location.

[0264] Optionally, a gesture in which a user pinches two or more of his or her fingers together at a specific location on a proximity-sensitive display associated with a specific content segment and then moves the two or more pinched fingers away from the specific location includes: a gesture in which a user pinches two or more of his or her fingers together at a specific location on a proximity-sensitive display associated with a specific content segment and then moves the two or more pinched fingers together to an edge of the proximity-sensitive display.

[0265] Optionally, a gesture in which a user pinches two or more of his or her fingers together at a specific location on a proximity-sensitive display associated with a specific content segment and then moves the two or more pinched fingers away from the specific location includes: a gesture in which a user pinches two or more of his or her fingers together at a specific location on a proximity-sensitive display associated with a specific content segment and then lifts the two or more pinched fingers away from the proximity-sensitive display.

[0266] Optionally, presenting the specific content segment on the proximity-sensitive display includes: presenting a scene including the specific content segment on the proximity-sensitive display.

[0267] Optionally, stopping presenting the specific content segment on the proximity-sensitive display includes: moving the specific content segment out of the scene.

[0268] Optionally, the method further includes: when presenting the specific content segment on the proximity-sensitive display, generating identification information for the specific content segment, and sending the identification information of the specific content segment to the wearable display system.

[0269] Optionally, the method also includes: when presenting the specific content segment on the proximity-sensitive display, generating data indicating the current position of the specific content segment relative to the proximity-sensitive display, and sending the data indicating the current position of the specific content segment to the wearable display system.

[0270] Optionally, the method may be performed by the computing device described immediately above.

[0271] A wearable computing system includes: a head-mounted display configured to be worn on a user's head; one or more cameras configured to capture images of an environment in front of the user; and one or more processors operably coupled to the head-mounted display and the one or more cameras and communicatively coupled to a computing device, the one or more processors configured to: identify a specific content segment being presented on a display of the computing device; receive one or more messages from the computing device indicating that presentation of the specific content segment is to be switched from the display of the computing device to the head-mounted display; in response to receiving the one or more messages from the computing device: determine a location in the environment in front of the user using one or more images captured by the one or more cameras; and present the specific content segment on the head-mounted display so as to be perceived by the user as being located at the determined location in the environment in front of the user.

[0272] Optionally, the wearable computing system is configured to operate in the same or similar manner as one or more of the wearable computing systems, wearable display systems, and / or image display devices described herein.

[0273] Optionally, the wearable computing system is configured to perform one or more operations in the method described immediately below.

[0274] A computer-implemented method includes: identifying a particular content segment being presented on a display of a computing device; receiving one or more messages from the computing device indicating that presentation of the particular content segment is to be switched from the display of the computing device to a head-mounted display; and in response to receiving the one or more messages from the computing device: obtaining one or more images of an environment in front of a user of the head-mounted display; identifying a location in the environment in front of the user based on the one or more obtained images; and presenting the particular content segment on the head-mounted display so as to be perceived by the user as being located at the determined location in the environment in front of the user.

[0275] Optionally, the display in the computing device on which the particular content segment is presented is a two-dimensional screen.

[0276] Optionally, presenting a specific content segment on a head-mounted display so as to be perceived by the user as being located at a determined location in an environment in front of the user includes: presenting a three-dimensional representation of the specific content segment on the head-mounted display so as to be perceived by the user as being located at the determined location in the environment in front of the user.

[0277] Optionally, the method further includes receiving identification information for the specific content segment from a computing device.

[0278] Optionally, identifying the specific content segment being presented on the display of the computing device includes: identifying the specific content segment being presented on the display of the computing device based on identification information received from the computing device.

[0279] Optionally, the method further comprises receiving data from a computing device indicating a current location of the particular content segment.

[0280] Optionally, the current position of the particular content segment corresponds to a current position of the particular content segment relative to a display of the computing device.

[0281] Optionally, the method may be performed by the wearable computing system described immediately above.

[0282] A system includes: a computing device configured to operate independently for a dedicated input controller associated with a wearable computing system, wherein the computing device includes: a proximity-sensitive display; and one or more processors operably coupled to the proximity-sensitive display and communicatively coupled to the wearable computing system, the one or more processors configured to: present a plurality of graphical control elements at a plurality of locations on the proximity-sensitive display, respectively, wherein the plurality of graphical control elements respectively correspond to a plurality of physical user input components of the dedicated input controller; monitor for touch input received at any one of the plurality of locations on the proximity-sensitive display; and in response to detecting that a touch input is received at any one of the plurality of locations on the proximity-sensitive display: discern, from the plurality of graphical control elements associated with the plurality of locations on the proximity-sensitive display, a particular graphical control element among the plurality of graphical control elements associated with a particular location among the plurality of locations on the proximity-sensitive display where the detected touch input is received; generate one or more messages based on the particular graphical control element; and send the one or more messages to the wearable computing system.

[0283] Optionally, the plurality of physical user input components in the dedicated input controller corresponding to the plurality of graphical control elements include one or more of the following: a “home” button, a “trigger” button, a “buffer” button, and a touch pad.

[0284] Optionally, the computing device is configured to operate in the same or similar manner as one or more of the computing devices and / or apparatus described herein.

[0285] Optionally, the computing device is configured to perform one or more operations in the method described immediately below.

[0286] A computer-implemented method includes: providing a graphical user interface of an application running on a computing device for output on a proximity-sensitive display of the computing device; presenting, through the graphical user interface, a plurality of graphical control elements, respectively, the plurality of graphical control elements corresponding to a plurality of physical user input components in a dedicated input controller associated with a wearable computing system; receiving data indicating touch input received at a specific location on the proximity-sensitive display; determining that the specific location where the touch input was received is associated with a specific graphical control element among a plurality of graphical control elements; and in response to determining that the specific location where the touch input was received is associated with the specific graphical control element: generating one or more messages based on the specific graphical control element; and sending the one or more messages to the wearable computing system.

[0287] Optionally, the method may be performed by the computing device described immediately above.

[0288] Additional and other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0289] The accompanying drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale and that elements having similar structures or functions are represented by the same reference numerals throughout all drawings. In order to better understand how to obtain the above and other advantages and objectives of various embodiments of the present disclosure, a more detailed description of the present disclosure briefly described above will be presented by reference to specific embodiments of the present disclosure. It should be understood that these drawings illustrate only typical embodiments of the present disclosure and, therefore, should not be considered limiting of its scope, and the present disclosure will be described and explained with the aid of additional features and details through the use of the accompanying drawings, in which:

[0290] Figure 1 Another image display system having an image display device according to some embodiments is shown.

[0291] Figure 2 Another image display system having an image display device according to other embodiments is shown.

[0292] Figure 3 Another image display system having an image display device according to other embodiments is shown.

[0293] Figure 4 Another image display system having an image display device according to other embodiments is shown.

[0294] Figure 5 An image display device that displays frames at multiple depth planes is shown.

[0295] Figure 6 An apparatus is shown having a touch sensitive screen with a designated area for allowing a user to provide input to an image display device.

[0296] Figures 7A-7F Shows Figure 6 Examples of different optional designated areas of a screen of a device in FIG.

[0297] Fig. 8A shows a response to a user touching Figure 6 An example of visual feedback generated in response to a portion of a designated area of ​​a screen of a device.

[0298] Figure 8B shows a response to the user touching with two fingers Figure 6 An example of visual feedback generated in response to a portion of a designated area of ​​a screen of a device.

[0299] Fig.9AIt shows that it can be Figure 6 Example of a keyboard displayed on a device in .

[0300] Fig. 9B It shows that it can be Figure 6 Another example of a keyboard displayed by a device in FIG.

[0301] Fig. 10A Shown in Figure 6 An example of a finger slide performed on a designated area of ​​a screen of a device in the image display apparatus and detectable by the device as a finger motion input for the image display device.

[0302] Fig. 10B Shown in Figure 6 Another example of a finger slide that is performed on a designated area of ​​a screen of a device and detectable by the device as a finger motion input for an image display apparatus.

[0303] Fig.11A Shown in Figure 6 An example of a finger touch movement performed on a designated area of ​​a screen of a device in the image display apparatus and detectable by the device as a finger motion input for the image display device.

[0304] Fig. 11B Shown in Figure 6 An example of a two-finger touch movement performed on a designated area of ​​a screen of a device in and detectable by the device as a finger motion input for an image display apparatus.

[0305] Fig. 12A Shown in Figure 6 An example of a two-finger pinching as a finger motion input for an image display device is performed on a designated area of ​​a screen of a device in and can be detected by the device.

[0306] Fig. 12B Shown in Figure 6 An example of a two-finger spread as a finger motion input for an image display device is performed on a designated area of ​​a screen of a device in and can be detected by the device.

[0307] Fig.13 Shown in Figure 6 An example of radial movement of two fingers as a finger motion input for an image display device is performed on a designated area of ​​a screen of a device in and can be detected by the device.

[0308] Fig.14 Shown in Figure 6 An example of a long touch hold performed on a designated area of ​​a screen of a device in and detectable by the device as a finger motion input for an image display apparatus.

[0309] Fig.15 Shown in Figure 6 An example of a tap that is performed on a designated area of ​​a screen of a device in and can be detected by the device as a finger motion input for an image display apparatus.

[0310] Fig.16A Shown in Figure 6 An example of a finger motion that is performed on an area of ​​a screen of a device that has been designated as a button and that can be detected by the device as an input for an image display apparatus.

[0311] Fig. 16B Shown by Figure 6 An example of a combination of different finger motion inputs detectable by an apparatus in as a combined input for an image display device.

[0312] Fig.17 Shows Figure 6 The feedback characteristics of the device in.

[0313] Fig.18 An algorithm or method for providing feedback is shown.

[0314] Fig.19A The device is shown in the face-up orientation. Figure 6 Examples of finger actions performed on a device in FIG. 1 , and examples of corresponding effects on displayed content.

[0315] Fig.19B The device is shown in the upright orientation. Figure 6 Examples of finger actions performed on a device in FIG. 1 , and examples of corresponding effects on displayed content.

[0316] Fig. 20 Methods according to some embodiments are shown.

[0317] Fig.21 Shows Figure 6 A device in which an application is downloaded in the form of a set of instructions from a server.

[0318] Fig. 22 A dedicated processing system is shown in accordance with some embodiments. DETAILED DESCRIPTION

[0319] Various embodiments of the present disclosure are directed to methods, apparatuses, and articles of manufacture for providing input to a head mounted video imaging device. Other objects, features, and advantages of the present disclosure are described in the detailed description, drawings, and claims.

[0320] Various embodiments are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and throughout the drawings, elements with similar structures or functions are represented by the same reference numerals. It should also be noted that the drawings are intended only to facilitate the description of the embodiments. They are not intended to serve as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, the illustrated embodiments do not need to have all the aspects or advantages shown. The aspects or advantages described in conjunction with a specific embodiment are not necessarily limited to that embodiment, and even if not shown as such or if not explicitly described as such, the above aspects or advantages may also be practiced in any other embodiment.

[0321] The following description relates to an exemplary VR, AR, and / or MR system in which the embodiments described herein may be practiced. However, it should be understood that the embodiments are also applicable to applications in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the exemplary examples disclosed herein.

[0322] Overview of Problems and Solutions

[0323] In some cases, a head-mounted image display device may have a dedicated input controller for allowing a user of the head-mounted image display device to provide input. The dedicated input controller may be specific to a specific model of the head-mounted image display device, and may be unique for each brand and model of the image display device. Such a dedicated input controller may allow a user of the head-mounted image display device to enter text by displaying a keyboard on the screen of the head-mounted image display device, and the user may enter text by interacting with the keyboard displayed on the screen of the head-mounted image display device using a dedicated input controller. This technique for entering text may be cumbersome and may not be ideal. In addition, the dedicated input controller may have limited voice and gesture control. In addition, if the dedicated input controller for a specific head-mounted image display device is lost, if the dedicated input controller is damaged, or if the user forgets to bring the dedicated input controller, there is no backup controller for the user to use. In addition, the dedicated input controller of the head-mounted image display device may not be able to send digital images, photos, and other media content between the user's media storage device and the head-mounted image display device. The dedicated input controller may also be unable to access application stores and content, and may be unable to receive notifications from the head mounted image display device or from a network when the head mounted image display device is not being worn by the user.

[0324] According to one or more embodiments described herein, a device with a touch screen is provided as an input device for allowing a user of a head-mounted image display device to enter input for the image display device, which input device is sometimes referred to herein as a "wearable computing system" or "wearable display system". The device, sometimes more simply referred to herein as a "computing device", can be used as a backup input device so that if a dedicated input controller of the head-mounted image display device is not available, the device can be used by the user to enter input instead. The device can also be used as a primary or sole input device for the image display device instead. The device can be used as a primary device for entering text, or alternatively, it can be used as a substitute for a dedicated input controller for entering text. When the user views the screen of the device through a transparent portion of the head-mounted image display device, entering text through the touch screen of the device is more convenient and efficient than displaying a keyboard on the viewing screen of the image display device when the user enters text using a dedicated input controller. Moreover, in an embodiment where the device is implemented using a user's mobile phone, the device can also facilitate the sending of digital images and photos and other media content between the user's phone and the image display device. In addition, in some embodiments, the device described herein allows a user of a head-mounted image display device to access an application store and content. Furthermore, in some embodiments, the apparatus described herein may receive a notification from the head-mounted image display device or from a network when the head-mounted image display device is not being worn by the user.

[0325] Figure 1-4 Various components of the image display system 100 in various embodiments are shown. The image display system 100 includes an image display device 101 and an apparatus 200 for providing input to the image display device 101. The apparatus 200 will be described in more detail below. The image display device 101 can be any of a VR device, an AR device, an MR device, or other types of display devices. The image display device 101 includes a frame structure 102 worn by a terminal user 50, a display subsystem 110 carried by the frame structure 102 so that the display subsystem 110 is positioned in front of the eyes of the terminal user 50, and a speaker 106 carried by the frame structure 102 so that the speaker 106 is positioned adjacent to the ear canal of the terminal user 50 (optionally, another speaker (not shown) is positioned adjacent to another ear canal of the terminal user 50 to provide stereo / formable sound control). The display subsystem 110 is designed to present a light pattern that can be comfortably perceived as an enhancement of physical reality to the eyes of the terminal user 50 with a high level of image quality and three-dimensional perception, and can present two-dimensional content. The display subsystem 110 presents the sequence of frames at a high frequency that provides the perception of a single coherent scene.

[0326] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display through which a user can directly view light from a real object via a transparent (or translucent) element. The transparent element, which is often referred to as a "combiner," superimposes light from the display on the user's view of the real world. To this end, the display subsystem 110 includes a partially transparent display. The display is positioned in the end user's 50 field of view between the end user's 50 eyes and the surrounding environment so that direct light from the surrounding environment is sent through the display to the end user's 50 eyes.

[0327] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, which is combined with direct light from the surrounding environment and sent from the display to the eyes of the user 50. The projection subsystem can be a fiber scanning based projection device, and the display can be a waveguide based display, where the scanned light from the projection subsystem is injected into the waveguide based display to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focus planes, and / or image layers stacked at multiple viewing distances or focus planes to represent a volumetric 3D object. The layers in the light field can be stacked closely enough together to appear continuous to the human visual subsystem (i.e., one layer is within the cone of confusion of the adjacent layer). Additionally or alternatively, even if the layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of the adjacent layer), image elements can be blended across two or more layers to increase the perceived continuity of the transition between layers in the light field. The display subsystem 110 can be monocular or binocular.

[0328] The image display device 101 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the eye position and inter-eye distance of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radios, and / or gyroscopes) or any combination of the foregoing. Many of these sensors operate based on the assumption that the frame 102 to which they are fixed is, in turn, substantially fixed to the user's head, eyes, and ears.

[0329] The image display device 101 may also include a user orientation detection module. The user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via a sensor coupled to the frame 102), and may predict the position of the head 54 of the end user 50 based on the position data received from the sensor. Detecting the instantaneous position of the head 54 of the end user 50 helps determine the specific real object that the end user 50 is looking at, thereby providing an indication of a specific virtual object to be generated with respect to the real object, and further providing an indication of the location where the virtual object is to be displayed. The user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensor.

[0330] The image display device 101 may also include a control subsystem that may take any of a variety of forms. The control subsystem includes a plurality of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs) (e.g., field PGAs (FPGAs)), and / or programmable logic controllers (PLUs).

[0331] The control subsystem of the image display device 101 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional database for storing three-dimensional scene data. The CPU may control the overall operation, while the GPU may render frames (i.e., convert the three-dimensional scene into a two-dimensional image) from the three-dimensional data stored in the three-dimensional database and store the frames in the frame buffer. One or more additional integrated circuits may control the reading and / or reading out of frames from the frame buffer and the operation of the image projection components of the control display subsystem 110.

[0332] The various processing components of the image display device 101 may be physically contained in a distributed subsystem. Figure 1-4 As shown, the image display device 101 includes a local processing and data module 130 that is operably coupled (e.g., via a wired lead or wireless connection 136) to the display subsystem 110 and the sensor. The local processing and data module 130 can be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 ( Figure 1 ), fixedly attached to a helmet or hat 56 ( Figure 2 ), removably attached to the torso 58 of the end user 50 ( Figure 3 ) or is configured to be removably attached to the hip 60 of the end user 50 in a belt coupling manner ( Figure 4The image display device 101 also includes a remote processing module 132 and a remote data repository 134 operably coupled (e.g., via wired leads or wireless connections 138, 140) to the local processing and data module 130, such that these remote modules 132, 134 are operably coupled to each other and can serve as sources for the local processing and data module 130.

[0333] The local processing and data module 130 may include a power efficient processor or controller and digital memory such as flash memory, both of which may be used to help process, cache and store data captured from the sensor and / or data acquired and / or processed using the remote processing module 132 and / or remote data repository 134, which may be transferred to the display subsystem 110 after such processing or retrieval. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository 134 may include a relatively large-scale digital data storage facility that is available through the Internet or other network configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130, allowing for full autonomous use from any remote module.

[0334] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wired or optical communications or one or more wireless interfaces or ports for providing wireless communications such as through RF, microwave, and IR. In some implementations, all communications may be wired, while in other implementations, all communications may be wireless. In further implementations, the choice of wired and wireless communications may be with Figure 1-4 Therefore, the specific choice of wired or wireless communication should not be considered limiting.

[0335] In some embodiments, the user orientation module is included in the local processing and data module 130, while the CPU and GPU are included in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be included in the local processing and data module 130. The 3D database may be associated with the remote data repository 134 or may be provided locally.

[0336] Some image display systems (e.g., VR systems, AR systems, MR systems, etc.) use multiple volume phase holograms, surface relief holograms, or light-guiding optical elements embedded with depth plane information to generate images that appear to come from various depth planes. In other words, a diffraction pattern or diffractive optical element ("DOE") can be embedded in a light-guiding optical element ("LOE"; for example, a planar waveguide) or embossed / engraved on the LOE so that collimated light (a light beam with a substantially planar wavefront) is substantially totally internally reflected along the LOE, intersecting the diffraction pattern at multiple locations and exiting toward the user's eyes. The DOE is configured so that light emitted from the LOE through it converges so that they appear to originate from a specific depth plane. Collimated light can be generated using an optical focusing lens ("condenser").

[0337] For example, a first LOE can be configured to transmit collimated light to the eye that appears to originate from an optically infinite depth plane (0 diopters). Another LOE can be configured to transmit collimated light that appears to originate from a distance of 2 meters (1 / 2 diopters). Yet another LOE can be configured to transmit collimated light that appears to originate from a distance of 1 meter (1 diopters). By using stacked LOE assemblies, it will be appreciated that multiple depth planes can be formed, wherein each LOE is configured to display an image that appears to originate from a particular depth plane. It should be appreciated that the stack can include any number of LOEs. However, at least N stacked LOEs are required to generate the N depth planes. In addition, RGB color images can be generated on N depth planes using N, 2N, or 3N stacked LOEs.

[0338] In order to present 3D virtual content to the user, the image display system 100 (e.g., VR system, AR system, MR system, etc.) projects images of the virtual content into the user's eyes so that the images appear to originate from various depth planes in the Z direction (i.e., orthogonally away from the user's eyes). In other words, the virtual content can not only change in the X and Y directions (i.e., in a 2D plane orthogonal to the central visual axis of the user's eyes), but also appear to change in the Z direction, so that the user can perceive that the object is very close or at an infinite distance or at any distance between them. In other embodiments, the user can perceive multiple objects at different depth planes at the same time. For example, the user can see a virtual dragon that appears to be running towards the user from an infinite distance. Alternatively, the user can simultaneously see a virtual bird at a distance of 3 meters from the user and a virtual coffee cup at an arm's length (about 1 meter) from the user.

[0339] A multi-plane focusing system creates a variable depth perception by projecting images on some or all of multiple depth planes at various fixed distances from the user's eyes in the Z direction. Figure 5It should be understood that the multi-plane focusing system can be used at a fixed depth plane 150 (e.g., Figure 5 Frames are displayed at six depth planes 150 as shown. Although the MR system may include any number of depth planes 150, an exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When virtual content is generated at one or more of the six depth planes 150, a 3D perception is formed so that the user perceives one or more virtual objects at different distances from the user's eyes. Assuming that the human eye is more sensitive to objects that are closer than objects that appear to be farther away, more depth planes 150 that are closer to the eye are generated, such as Figure 5 In other embodiments, the depth planes 150 may be placed at equal distances from each other.

[0340] The depth plane position 150 may be measured in diopters, which is a unit of optical power that is equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopters away, depth plane 2 may be 0.3 diopters away, depth plane 3 may be 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate depth planes 150 at other distances / diopters. Thus, when virtual content is generated at strategically placed depth planes 150, the user is able to perceive three-dimensional virtual objects. For example, the user may perceive the first virtual object displayed in depth plane 1 as being close to him, while another virtual object appears to be at depth plane 6 at infinity. Alternatively, the virtual object may be first displayed at depth plane 6, then displayed at depth plane 5, and so on, until the virtual object appears to be very close to the user. It should be understood that the above examples are significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be focused at a specific focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes may be generated at various cross-sections of the coffee cup, giving the user a highly granular 3D view of the coffee cup.

[0341] In some embodiments, the image display system 100 (e.g., a VR system, an AR system, an MR system, etc.) can be used as a multi-plane focusing system. In other words, all six LOEs can be illuminated simultaneously, thereby generating images that appear to originate from six fixed depth planes in rapid succession, where the light source quickly delivers image information to LOE 1, then LOE 2, then LOE 3, and so on. For example, a portion of the desired image can be injected at time 1, which includes an image of the sky at optical infinity, and the LOEs that maintain light collimation (e.g., Figure 5Then, an image of a closer branch can be injected at time 2, and a depth plane configured to appear to originate from 10 meters away (e.g., Figure 5 Then, the pen image can be injected at time 3, and the LOE configured to generate an image that appears to originate from a depth plane 1 meter away can be utilized. Such examples can be repeated in extremely fast timing (e.g., at 360 Hz) so that the user's eyes and brain (e.g., visual cortex) perceive the inputs as all parts of the same image.

[0342] The image display system 100 can project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) to generate images for a 3D experience / scene. As used in this application, a "light beam" includes, but is not limited to, a directional projection of light energy (including visible and invisible light energy) radiated from a light source. Generating images that appear to originate from various depth planes complies with the vergence and accommodation of the user's eyes for the image, and minimizes or eliminates the vergence-accommodation conflict.

[0343] like Figure 1-4 As shown, the image display system 100 includes an apparatus 200 for providing input to the image display device 101. The apparatus 200 allows a user 50 of the image display device 101 to enter user input while the user 50 is viewing content displayed by the image display device 101. In one embodiment, the apparatus 200 may be a handheld device (e.g., a mobile phone (e.g., a smart phone), a tablet computer, an IPAD, a mini tablet computer, etc.) configured to interact with the image display device 101. Figure 1-4 In the example shown, the apparatus 200 interacts with the image display device 101 by communicating with the processing module 130 of the image display device 101. The communication between the apparatus 200 and the processing module 130 of the image display device 101 can be achieved via a wireless connection or a wired connection. In other embodiments, the apparatus 200 can also communicate with a remote processing module 132 and / or a remote data repository 134 via a wireless connection.

[0344] Figure 6An example of an apparatus 200 is shown, the apparatus 200 is used for use with an image display device 101, the image display device 101 being configured to be worn by a user 50. Specifically, the apparatus 200 is configured as an input device / controller for allowing the user 50 to provide user input using his / her finger while the user 50 is viewing an image displayed by the display device 101. In some cases, the apparatus 200 may be considered as part of the image display system 100. The apparatus includes a screen 202 and a processing unit 204. The processing unit 204 is configured to designate a first area 206 of the screen 202 to sense finger motions of the user 50. The processing unit 204 is configured to detect different finger motions performed on the designated first area 206, thereby allowing the designated first area 206 to function like a touch pad. The processing unit 204 is also configured to generate an electronic signal based on the finger motion of the user 50 sensed by the designated first area of ​​the screen 202 of the apparatus 200 to cause a change in the content displayed by the image display device 101. The electronic signal may represent an identification of the detected finger motion, and / or may represent a command generated by the processing unit 204 based on the detected finger motion.

[0345] In the illustrated embodiment, the device 200 is a handheld device. By way of non-limiting example, the handheld device may be a cellular phone, a smart phone, a personal digital assistant (PDA), or a tablet computer.

[0346] like Figure 6 As shown, the screen 202 of the device 200 has a touch-sensitive area 210, and the designated first area 206 is smaller than the total area of ​​the touch-sensitive area 210. Therefore, the processing unit 204 is configured to ignore input generated by the user 50 using a portion of the touch-sensitive area 210 that is not a portion of the designated first area 206.

[0347] The size and shape of the first area 206 may be predefined or may be selectively configured by the user 50. In other embodiments, the first area 206 may have other shapes instead of Figure 6 The square shape shown in . Figures 7A-7F Examples of different first regions 206 that may be specified by processing unit 204 are shown. Fig. 7A 200, wherein the long side of the first region 206 is parallel to the short side of the device 200, and the short side of the first region 206 is parallel to the long side of the device 200. In other embodiments, as Figure 7BAs shown, the first area 206 may have a rectangular shape, wherein the long sides of the first area 206 are parallel to the long sides of the device 200, and the short sides of the first area 206 are parallel to the short sides of the device 200. In other embodiments, the size of the first area 206 may be smaller (or larger) than the example shown. For example, in other embodiments, the first area 206 may have a small size ( Figure 7C In another embodiment, the first region 206 may have a circular shape ( Fig.7D ), hexagonal shape ( Fig. 7E ) or any other polygonal shape.

[0348] In some embodiments, the first area 206 may have a shape corresponding to an area of ​​a display screen of the image display device 101. For example, if the screen of the image display device 101 has an aspect ratio of 4:3, the first area 206 designated by the processing unit 204 may also have an aspect ratio of 4:3 ( Figure 7F In one embodiment, the processing unit 204 of the apparatus 200 is configured to determine the shape of the display screen area of ​​the image display device 101, and determine the shape of the first area 206 based on the shape of the display screen area of ​​the image display device 101. For example, the processing unit 204 may determine that the shape of the first area 206 is the same as the shape of the display screen area of ​​the image display device 101.

[0349] In some embodiments, the processing unit 204 may be configured to determine the size of the touch-sensitive area 210 and determine the size and / or shape of the first area 206 designated to sense finger motions of the user 50. For example, the processing unit 204 may determine that the width of the entire touch-sensitive area 210 of the device 200 is 2.5 inches and the length is 5.5 inches. In such a case, the processing unit 204 may then determine the width of the first area 206 to be a constant k1 (e.g., 0.9) times the width of the touch-sensitive area 210 (in the above example, it would be 0.9x2.5=2.25 inches). Similarly, the processing unit 204 may determine the length of the first area 206 to be a constant k2 (e.g., 0.5) times the length of the touch-sensitive area 210 (in the above example, it would be 0.5x5.5=2.75 inches).

[0350] In other embodiments, the processing unit 204 may determine the brand and model of the device 200 and determine the size and / or shape of the first area 206 to be designated to sense finger motions of the user 50. For example, a brand and model of a device 200 having a relatively large screen size may be designated a relatively large first area 206 compared to another brand and another model of a device 200 having a relatively small screen size.

[0351] After the first area 206 has been designated for sensing the finger motion of the user 50, the user 50 can then use one or more fingers to input input for the image display system 100 using the designated first area 206 of the device 200. For example, the user 50 can move a finger within the designated first area 206 to move an object displayed by the image display device 101. The object can be any object, such as a cursor, an image, a photo, a window, a frame, an application page, etc. As another example, the user 50 can move two fingers in the same sliding direction within the designated first area 206 to move an object displayed by the image display device 101. As another example, the user 50 can perform a contraction or expansion action within the designated first area 206 to change the size of the object (e.g., reduce the size by contraction, increase the size by expansion). As another example, the user 50 can perform a tapping action (e.g., single click, double click, etc.) within the designated first area 206 to select an object, deselect an object, grab an object, cancel grabbing an object, etc. Thus, the apparatus 200 is configured as an input device / controller for allowing the user 50 to provide input for interacting with content displayed by the image display device 101 .

[0352] In the first operation mode, when the user 50 is using the device 200 to generate input for the image display system 100, the device 200 may not provide a visual display on the screen 202. When the designated first area 206 of the screen 202 is sensing the finger action of the user 50, the designated first area 206 of the screen has no displayed object. In such a case, the device 200 is configured like a touch pad. However, unlike the touch pad, the device 200 itself can be a mobile or handheld device. In some embodiments, when in the first operation mode, the presentation of content on the screen 202 by the device 200 can be at least partially degraded or disabled. More specifically, when in the first operation mode, in such an embodiment, the device 200 can provide less content for visual display on the screen 202, content with reduced brightness and / or contrast for visual display on the screen 202, reduce the brightness level of the screen 202, or take one or more other actions to reduce the display function to save power. In some examples, when in the first operation mode, the device 200 can communicate the boundaries of the touch pad and / or other parts of the user interface to which the touch pad belongs by using one or more feedback components (e.g., tactile feedback components, speakers, etc.). In these examples, when in the first operation mode, the device 200 can use one or more feedback components to communicate one or more parts of such a user interface, instead of using the screen 202 to visually display the user interface or in addition to it. In some embodiments, when the device 200 is operated in the first mode, a representation of such a user interface or a portion thereof can be displayed by the image display device 101. In at least some of these embodiments, the relative position of the touch input provided on the touch screen 202 can also be visually represented to the user by the image display device 101. In this way, the user can effectively interact with the screen 202 while looking elsewhere. In some of these examples, when in the first operation mode, instead of or in addition to using the screen 202 of the device 200 to visually display the user interface or a limited version thereof (e.g., a version of the user interface that is at least partially degraded or disabled), the image display device 101 can be used to display such a user interface or a representation of a portion thereof.

[0353] In the second operating mode, processing unit 204 may be configured to operate screen 202 to display content while allowing screen 202 to receive finger motion input from user 50. In some embodiments, the amount, brightness, and / or contrast of content provided by device 200 for visual display on screen 202 when in the second operating mode may be greater than the amount, brightness, and / or contrast of content provided by device 200 for visual display on screen 202 when in the first operating mode. Similarly, in some examples, device 200 may operate screen 202 at a higher brightness level in the second operating mode than in the first operating mode. For example, processing unit 204 may operate screen 202 to display a grid of dots 220 in a designated first area 206 of screen 202, such as Figure 6 The grid of points 220 allows user 50 to see where designated first area 206 is located, so that user 50 can more accurately position his / her finger. In some cases, processing unit 204 may also be configured to change the characteristics of one or more of points 220 in response to user 50 touching a portion of one or more of displayed points 206 in designated first area 206 of screen 202 ( Figures 8A-8B ). Specifically, Fig. 8A An example of visual feedback 230 generated by processing unit 204 in response to user 50 touching a portion of designated area 206 of screen 202 of device 200 with one finger is shown. Figure 8B An example of visual feedback 230 generated by processing unit 204 in response to user 50 touching a portion of designated area 206 of screen 202 of device 200 with two fingers is shown. This allows user 50 to see that the input generated by his / her finger action is successfully received by processing unit 204. In the above example, visual feedback 230 includes a circle and a change in the spacing of points 220 near the touch location. In other embodiments, visual feedback 230 may include only a circle without a change in the spacing of points 220. In other embodiments, instead of a circular shape, visual feedback 230 may include an object (of any shape) generated by processing unit 204 that is positioned at the location touched by user 50 in screen 202. In other embodiments, visual feedback 230 may not include any object overlapping with point 220. In such a case, visual feedback 230 may be a change in the color of the point at the location where user 50 touches screen 202, and / or a change in the shape and / or size of the point.

[0354] In some embodiments, the horizontal spacing of points 220 may be the same as the vertical spacing of points 220. In other embodiments, the horizontal spacing of points 220 may be different than the vertical spacing of points 220. For example, the horizontal spacing of points 202 may be greater or less than the vertical spacing of points 220.

[0355] It should be noted that in the second mode, the screen of the device 200 is not limited to displaying the dots 220, and other content may be displayed in other embodiments. For example, in other embodiments, when in the second mode, the screen of the device 200 may display a plurality of objects (e.g., photos, videos, icons, etc.) for the user 50 to select using the device 200, a text box for the user 50 to enter text, a drawing "pad" for allowing the user 50 to draw things, a tab for the user 50 to select, an application graphic for the user 50 to interact with, etc. In some embodiments, a polygonal shape may be displayed in the designated first area 206 of the screen 202. For example, in these embodiments, the device may display a solid polygonal shape, an outline of a polygonal shape, or both in the designated first area 206 of the screen 202. In some examples, the geometry of such a polygonal shape may correspond to the geometry of the designated first area 206, such that one or more boundaries of the polygonal shape may be respectively consistent with one or more boundaries of the designated first area 206.

[0356] In some embodiments, the device 200 can switch between the first mode and the second mode. For example, the device 200 can switch between the first mode and the second mode based on the command received from the image display device 101. In some embodiments, the image display device 101 can analyze the data output from one or more devices in its sensing device to determine whether the device 200 is in the user's field of view. In some examples, such one or more sensing devices of the image display device 101 may correspond to one or more forward cameras. For example, the image display device 101 can analyze the image captured by its camera to determine whether the device 200 is displayed in any one of the images (and therefore located in the user's field of view). If the device 200 is not detected by any one of the cameras, then in response, the image display device 101 can then indicate that the device 200 operates in the first mode. On the other hand, if the device 200 is detected by the camera of the image display device 101, then in response, the image display device 101 can then indicate that the device 200 operates in the second mode. This feature is advantageous because it can sometimes provide power saving benefits when the user 50 does not watch the screen of the device 200. Thus, if the user 50 is not viewing the apparatus 200 through the screen of the image display device 101, the screen of the apparatus 200 may operate in a first mode (displaying no content or a degraded set of content), and if the user 50 is viewing the apparatus 200 through the screen of the image display device 101, the screen of the apparatus 200 may operate in a second mode (displaying the entire content). In some examples, the image display device 101 may utilize, in addition to one or more of its forward-facing cameras, one or more of the sensing devices to determine whether the apparatus 200 is within the user's field of view. For example, the image display device 101 may determine whether the apparatus 200 is within the user's field of view based on data output from one or more proximity sensors, an inward-facing eye tracking camera, or other sensing devices of the image display device 101 that are capable of outputting data indicating whether the user is wearing the image display device 101. In implementations where the image display device 101 is a virtual reality (VR) headset or other wearable computing device that physically occludes some portion of the user's environment when worn by a user, it is particularly advantageous that the image display device 101 can output an indication device 200 to switch between first and second operating modes based on data from one or more such sensing devices.

[0357] In addition, in some embodiments, each unit length Lh of the horizontal movement input performed at the first area 206 may result in a corresponding horizontal movement amount Mh of the object displayed by the image display device 101. Similarly, each unit length Lv of the vertical movement input performed at the first area 206 may result in a corresponding vertical movement amount Mv of the object displayed by the image display device 101. The horizontal movement efficiency may be represented by a ratio Mh / Lh, and the vertical movement efficiency may be represented by a ratio Mv / Lv. If the ratio Mh / Lh is 1.0, this means that one unit of horizontal movement input at the designated first area 206 will result in the same unit of horizontal movement occurring for the object displayed by the image display device 101. If the ratio Mh / Lh is greater than 1.0, this means that one unit of horizontal movement input at the designated first area 206 will result in more than one unit of horizontal movement occurring for the object displayed by the image display device 101. If the ratio Mh / Lh is less than 1.0, it means that one unit of horizontal movement input at the designated first area 206 will result in less than one unit of horizontal movement occurring for the object displayed by the image display device 101. Similar concepts apply to the vertical direction. In some embodiments, the size of the screen of the device 200 may be smaller than that of the screen of the image display device 101. In such a case, it may be desirable to configure the specified first area 206 so that Mh / Lh>1, and so that Mv / Lv>1. This allows the user 50 to operate a relatively small first area 206 on the device 200 to cover a larger screen area in the image display device 101. In other embodiments, the size of the screen of the device 200 may be larger than that of the screen of the image display device 1011. In such a case, it may be desirable to configure the specified first area 206 so that Mh / Lh<1, and so that Mv / Lv<1. This allows the user 50 to operate a relatively large first area 206 on the device 200 to cover a smaller screen area in the image display device 101. In other embodiments, regardless of the relative screen sizes of the device 200 and the image display device 101, the movement efficiency (Mh / Lh and Mv / Lv) of the specified first area 206 may be arbitrarily set to any value.

[0358] In some embodiments, the ratio Mh / Lh may be equal to the ratio Mv / Lv. This means that for inputs inputted using the specified first region 206, the horizontal movement efficiency is the same as the vertical movement efficiency. In other embodiments, the ratio Mh / Lh may be higher than the ratio Mv / Lv. This means that for inputs inputted using the specified first region 206, the horizontal movement efficiency is higher than the vertical movement efficiency. This allows the user 50 to cover more horizontal spaces in the display screen of the image display device 101 by moving through less horizontal spaces (compared with the vertical spaces) in the specified first region 206. In other embodiments, the ratio Mh / Lh may be less than the ratio Mv / Lv. This means that for inputs inputted using the specified first region 206, the horizontal movement efficiency is lower than the vertical movement efficiency. This allows the user 50 to cover more vertical spaces in the display screen of the image display device 101 by moving through less vertical spaces (compared with the horizontal spaces) in the specified first region 206.

[0359] In some embodiments, Mh / Lh may be based on the width of the screen of the image display device 101, and Mv / Lv may be based on the height of the screen of the image display device 101. Therefore, Mh / Lh and Mv / Lv may be based on the aspect ratio or shape of the screen of the image display device 101. For example, if the screen of the image display device 101 has an aspect ratio (width to height ratio) of 4:3, Mh / Lh may be 1.33 times that of Mv / Lv. When these movement efficiencies are applied to the designated first area 206 having a square shape, for example, a one-unit horizontal movement input by the user will cause the object displayed in the image display device 101 to move 1.33 times compared to a one-unit vertical movement. Therefore, even though the designated first area 206 (having a square shape in this example) may not correspond to the shape of the screen of the image display device 101 (having an aspect ratio of 4:3 in the example), the movement efficiency of the designated first area 206 can be selected or determined so that the user 50 can use the designated first area 206 to cover all spaces horizontally and vertically.

[0360] In other embodiments, the horizontal movement efficiency and / or vertical movement efficiency of the specified first area 206 can be variable. In particular, the horizontal movement efficiency and / or vertical movement efficiency of the specified first area 206 can be a function of the input position in the specified first area 206, and / or a function of the cursor / pointer position in the screen of the image display device 101. For example, when the input generated at the first area 206 corresponds to the center of the field of view or the center of the first area 206, the horizontal movement efficiency can have a first value, and when the input generated at the first area 206 corresponds to the position at the periphery of the field of view or at the periphery of the first area 206, it can have a second value (higher than the first value). When his / her finger approaches the left / right edge or the field of view of the first area 206, this feature may be desired. The scrolling speed may appear faster because the cursor "moves more" through the content. In other embodiments, the second value may be lower than the first value.

[0361] In one or more embodiments described herein, the horizontal movement efficiency and / or the vertical movement efficiency of the first region 206 may be predetermined, may be determined by the processing unit 204 , and / or may be configurable by a user 50 of the device 200 .

[0362] Return to Figure 6 In some embodiments, the processing unit 204 may also be optionally configured to designate the second area 240 of the screen 202 as a first button 242. In the example shown, the first button 242 is a "home" button. In other embodiments, the first button 242 may be other types of buttons for performing other types of functions.

[0363] In addition, the processing unit 204 can optionally be configured to designate the third area 250 of the screen as a second button 252. In the example shown, the second button 252 is a "toggle" button. The toggle button can allow the user to perform a selection function, and / or change the function of a cursor or pointer. In other embodiments, the second button 252 can be other types of buttons for performing other types of functions.

[0364] In addition, the processing unit 204 can optionally be configured to designate the fourth area 260 of the screen as a third button 262. In the example shown, the third button 262 is a "buffer" button. The buffer button is a multi-function button depending on the software / operating system of the device 200. In some cases, the buffer button can allow the user 50 to control the volume, open pages, change magnification, etc. In other embodiments, the third button 262 can be other types of buttons for performing other types of functions.

[0365] In other embodiments, the processing unit 204 may designate more than three areas of the screen as corresponding input areas (eg, a pad area for receiving finger movement input, a button, a control, etc.).

[0366] In addition, in some embodiments, any control of the device 200 may be operable to generate input for the image display device 101. For example, in some embodiments, the volume control of the device 200 may be utilized to control the speaker of the image display device 101. In one embodiment, the processing unit 204 is configured to detect the actuation of the volume control at the device 200. In response to the detected actuation, the processing unit 204 then provides a corresponding control signal (e.g., a volume up signal, a volume down signal, a mute, etc.) to be sent to the image display device 101. The transmission of the control signal may be implemented wirelessly or by a cable. Then, the image display device 101 operates its speaker according to the control signal provided by the device 200.

[0367] In addition, if Figure 6As shown, in some embodiments, the processing unit 204 may provide an "image capture" button 274 that allows the user 50 to capture an image presented by the screen of the image display device 101. The image may include content displayed by the screen of the image display device 101, and / or content in the surrounding environment viewed through the screen of the image display device 101. In one implementation, the processing unit 204 is configured to detect the pressing of the "image capture" button 274, and in response, generate a control signal (for sending to the image display device 101) so that the image display device 101 performs a screenshot function to capture a screenshot of the content displayed by the image display device 101. Alternatively or additionally, the processing unit 204 may generate a control signal to cause the image display device 101 to operate its camera to take a picture of the environment viewed by the camera. The image of the displayed content and the image of the environment may be combined by the processing unit 204 and / or the processing unit 130 to form a composite image. In some cases, the "image capture" button 274 may also be pressed to capture an image of the content displayed by the device 200. The image displayed by the device 200 may be an artificially generated graphic and / or a camera image captured by a camera of the device 200. For example, the processing unit 204 may detect the pressing of the "image capture" button 274, and in response, execute a screenshot function to capture a screenshot of the content displayed by the device 200. In addition, in other embodiments, the "image capture" button 274 may be used to capture a video. For example, the "image capture" button 274 may be pressed and held for a certain duration. The processing unit 204 detects that there is a long hold on the "image capture" button 274, and operates the camera of the device 200 to start recording a video. The user 50 may long hold the "image capture" button 274 again to stop the recording of the video. Therefore, as long as the processing unit 204 detects that there is a second long hold on the "image capture" button 274, the processing unit 204 operates the camera to stop the recording of the video. It should be noted that other techniques for generating a video using the device 200 may be employed in other embodiments, and the "image capture" button 274 is merely an example of a control that may be used to generate a video.

[0368] In one or more embodiments, the image or video captured by the device 200 can be sent by the device 200 to a network (e.g., a cloud network) for storage. In such a case, the image display device 101 can obtain the stored image or video from the network (e.g., a cloud network) to display on its screen. Alternatively, the image or video captured by the device 200 can be sent to the image display device 101 by a short-range network (e.g., a Bluetooth network, Wi-Fi, etc.). In addition, in some embodiments, the device 200 can be configured to obtain media content (e.g., photos, pictures, videos, etc.) from the network. For example, another user can upload media content to a cloud network, and the user 50 can obtain such media content by accessing the cloud network via an account associated with the device 200 and / or the image display device 101.

[0369] In some embodiments, buttons 242, 252, 262 are disabled when user 50 is using the designated first area 206 (simulated touchpad area) of screen 202. This can have the benefit of preventing user 50 from accidentally touching and activating any of these buttons. In other embodiments, one or more of buttons 242, 252, 262 are not disabled when user 50 is using the designated first area 206 of screen 202, thereby allowing user 50 to perform simultaneous finger actions using the first designated area 206 (simulated touchpad area) and any of buttons 242, 252, 262.

[0370] like Figure 6 As shown, the processing unit 204 may also be optionally configured to designate another area 270 of the screen 202 as a keyboard activation button 272, and wherein the processing unit 204 is configured to operate the screen 202 to display the keyboard 274 (in response to the user 50 touching the designated second area 270 of the screen 202 where the keyboard activation button 272 is located. Fig.9A ). Fig. 9B Another example of a keyboard 274 is shown that may be displayed by screen 202. The displayed keyboard 274 allows user 50 to enter text, numbers, symbols, emoticons, messages, and the like.

[0371] Return to Figure 6, the processing unit 204 is configured to perform gesture recognition to identify finger actions performed on any or a combination of the designated areas 206, 240, 250, 260. In one embodiment, the device 200 includes a buffer, and when the user 50 enters input using finger actions performed on the designated area, the device 200 generates touch input data corresponding to the touch position on the screen 202. The processing unit 204 is configured to add the touch input data to the buffer and perform gesture recognition using the input data stored in the buffer. In some embodiments, the input data may include position data representing the corresponding touch position in the designated first area 206. The input data may also include corresponding timing data representing the corresponding time when the corresponding position is touched by the user 50. If the user 50 touches any of the buttons 242, 252, 262, the corresponding input data generated may include an identification of the button being touched, the timing of the touch action, and the duration of the button being touched.

[0372] In the illustrated embodiment, the processing unit 204 is configured to perform gesture recognition using the input data to identify a corresponding command for the gesture being performed. For example, if the processing unit 204 determines from the input data that the user 50 is performing a sliding motion, the processing unit 204 can then identify "move" as a command corresponding to the user's finger motion. As another example, if the processing unit 204 determines from the input data that the user 50 is performing a pinching motion, the processing unit 204 can then identify "size reduction" as a command corresponding to the finger motion.

[0373] It should be noted that the processing unit 204 is not limited to detecting the above-mentioned gestures or finger movements, and the processing unit 204 can detect other finger movements of the user 50. By way of non-limiting example, the processing unit 204 can be configured to detect a finger slide (e.g., up / down slide ( Fig. 10A )), left / right slide( Fig. 10B ), two-finger slide, one-finger touch to move ( Fig.11A ), two-finger touch to move ( Fig. 11B ), pinch two fingers( Fig. 12A ), two-finger spread( Fig. 12B ), move two fingers radially ( Fig.13 ), long touch hold( Fig.14 ), tap ( Fig.15), double-click, etc. In one embodiment, the processing unit 204 is configured to distinguish between a long hold action and a tap action based on the duration of contact of the user's finger with the screen 202. For example, if the touch duration is 0.7 seconds or less, the processing unit 204 can determine that the finger action is a tap action. On the other hand, if the touch duration is 1 second or more, the processing unit 204 can determine that the finger action is a long hold action. In addition, the processing unit 204 can detect finger actions performed using other specified areas of the screen 202, such as the area where the "home" button is located, the area where the "trigger" button is located, and the area where the "buffer" button is located. In some embodiments, the function of each of one or more of the "home", "trigger" and "buffer" buttons described herein may correspond to the function of each of one or more of the "home", "trigger" and "buffer" buttons described in U.S. Patent Application No. 15 / 965,702, which is incorporated herein by reference in its entirety. Similarly, in some examples, the functionality of the touchpad described herein may correspond to the functionality described in U.S. Patent Application No. 15 / 965,702, which is incorporated herein by reference in its entirety. As such, the device 200 may serve as an adequate replacement for a dedicated input controller. Fig.16A 2 shows an example of a finger action (e.g., long hold, tap, etc.) that can be performed on the “trigger” button and detected by the processing unit 204. In addition, the processing unit 204 can be configured to detect a combination of finger actions performed on different designated areas of the screen 202. For example, Fig. 16B As shown, the user 50 can hold the "trigger" button for a long time, and in doing so, can also use the first designated area 206 of the screen 202 to perform a touch movement. The processing unit 204 is configured to detect these two finger actions at the same time, and can regard them as a combined input for the image display device 101. For example, the processing unit 204 can determine that the combination of the above finger actions is to create a command for moving the selected object. The processing unit 204 can then send this command signal to the image display device to move the selected object. The processing unit 204 is configured to generate corresponding commands for the above different finger actions or combinations of finger actions performed on one or more designated areas in the screen 202.

[0374] In some embodiments, when the user 50 performs a certain finger action on a designated area of ​​the screen 202, the finger of the user 50 may unintentionally leave the screen 2022. To address this situation, the processing unit 204 may be configured to determine that a certain finger action (e.g., a sliding action, etc.) has ended after receiving a number of continuous samples exceeding a predetermined threshold, indicating that the user 50 has not touched any position on the screen 202. For example, the processing unit 204 may wait until it has received three (or other number of) consecutive empty samples before determining that such a sliding action has ended. In this example, the processing unit 204 may receive a series of touch input samples, followed by two consecutive empty samples (e.g., because the user's finger did not touch the screen 202 unintentionally), followed by a series of touch input samples. In such a case, the processing unit 204 will still process the entire sample sequence as if it were part of a continuous sliding action. This allows the user 50 to correct his / her action and complete the finger action without requiring the user to re-perform the previously performed action. The above features are advantageous because they help filter out noise and increase system robustness to measurement or sensing errors.

[0375] In some embodiments, the processing unit 204 is configured to send the identified command to the image display device 101 (e.g., to the processing module 130). The processing module 130 then determines what content to display to the user 50 for viewing based on the received command determined by the processing unit 204. In some cases, the same command can result in different effects on the content displayed by the image display device 101, depending on the specific application that the user 50 is using and / or the specific content that the user 50 is viewing. For example, if the user 50 is viewing a page that allows the user 50 to browse different pages of objects, the "move" command can cause the scrolling of different pages of objects. On the other hand, if the user 50 is in an application that allows objects to be moved, the "move" command can cause the object to move from one position to another.

[0376] As shown in the above embodiments, the device 200 is advantageous and improves the technical field of head-mounted image display devices in several aspects. First, the device 200 can be used as a backup input device (for a head-mounted image display device) so that if the dedicated input controller of the head-mounted image display device is not available (for example, if the dedicated input controller is lost, damaged, runs out of battery, etc.), the device 200 can be used instead by the user 50 to input input for the head-mounted image display device. Many users of image display devices already carry smartphones whenever they enter. Therefore, implementing the device 200 as an input controller for using a smartphone will increase the chances that the user has an input controller for the image display device. The device can also be used as the primary or sole input device for the image display device, so that the image display device may not need to be with a dedicated input controller.

[0377] As mentioned, many users of the image display device already carry a smartphone or other type of mobile device whenever they enter. Therefore, implementing the apparatus 200 as an input controller using a smartphone or any other type of mobile device will improve the portability and mobility of the image display device, because the user can use the image display device 101 anywhere (even without a dedicated input controller for the image display device 101), as long as the user has a mobile device. Backup and supplemental input support can also be implemented by the apparatus 200, because most users already carry a mobile device whenever they enter.

[0378] The device 200 can be used as a main device for inputting text, or alternatively, can be used as a substitute for a dedicated input controller for inputting text. When the user views the screen of the device through the transparent portion of the head-mounted image display device, it is more convenient and efficient to enter text through the touch screen of the device than to display a keyboard on the viewing screen of the image display device when the user uses a dedicated input controller to enter text. Although it is known that smart phones implement keyboards, the concept of using smart phones to implement keyboards in the context of providing input for head-mounted image display devices is considered to be novel and non-intuitive. This is because such a solution will require the user 50 of the head-mounted image display device 101 to view the screen 202 of the device 200 (where the keyboard is displayed) through the display screen of the image display device 101. Therefore, the solution will require the user to move the focus from the display screen of the image display device 101 to the screen 202 of the device 200 (which is visible through the display screen of the image display device 101). However, it is believed that despite the shift in visual focus, in some cases, the touch screen keyboard implemented as an input device of the image display device 101 on the device 200 may be more comfortable for some users.

[0379] Furthermore, in embodiments where the apparatus 200 is implemented using a user's cellular phone (e.g., a smart phone), the apparatus 200 may also facilitate the transfer of digital images and photos and other media content between the user's phone and the image display device 101. For example, pictures taken by the camera of the image display device 101 may be transferred to the user's apparatus 200, and pictures taken by the camera of the apparatus 200 may be transferred to the image display device 101. Apparatus that provides both content and control input by the user for a head mounted image display device is considered unique.

[0380] In addition, in some embodiments, the apparatus 200 described herein allows a user of a head-mounted image display device to access application stores and content through a network such as the Internet, a Bluetooth network, etc. Since most smartphones already have multiple network interfaces for multiple types of network connections (e.g., Wi-Fi connections, Bluetooth connections, connections to cellular towers, etc.), using a smartphone (or other type of portable network device) to implement the apparatus 200 will have the benefit of allowing the user to obtain media content and other information from multiple sources through different types of connections. For example, the apparatus 200 implemented as an input device for the image display device 101 can also access an application store for obtaining applications that can be used with the image display device 101. As another example, the apparatus 200 implemented as an input device for the image display device 101 can also obtain media content from another mobile device (e.g., via email, text messaging, airdrop, etc.) on a web page (Web). In some embodiments, the apparatus 200 can also allow the user 50 to control account settings for the image display device 101 and manage the image display device 101 through a user interface displayed on the screen 202 of the apparatus 200.

[0381] In addition, in some embodiments, even when the head-mounted image display device 101 is not worn by the user 50, the device 200 described herein can also receive notifications (for the image display device 101) from a network (e.g., the Internet). For example, the image display device 101 can push different notifications to the device 200. The push of notifications can be performed directly by the image display device 101, or indirectly through another network device (such as a component in a cloud network). By way of non-limiting example, the notification received by the device 200 can be a calendar notification, an advertising notification, a social media notification, an operation notification about the operation of the image display device 101 (e.g., battery status, storage level, update notification, etc.), or any combination of the foregoing. In addition, in some embodiments, a third party communicating with the image display device 101 can provide notifications to the image display device 101. In such a case, the image display device 101 can forward the notification to the device 200 so that the user 50 can see the notification even if the user 50 does not wear the image display device 101. The notification feature is advantageous because it allows functions such as calendar management to be more practical, and timely reminders and real-time updates for the image display device 101 can be received by the user via the apparatus 200 .

[0382] Sensory feedback

[0383] In some embodiments, the processing unit 204 may optionally be configured to operate a feedback component in the device 200 in response to a finger action of the user 50. Such a feature is advantageous because it allows the user 50 to know that his / her finger is reaching or crossing the boundary of the first area 206. Specifically, when the user 50 is viewing the content displayed by the image display device 101, the user 50 may not be viewing the device 200. In this way, the user 50 may not visually notice that his / her finger has moved out of the boundary of the first area 206 (which has been designated to receive finger action input). The feedback feature solves this technical problem and provides a technical improvement of the image display system 100 by notifying the user 50 of the image display system 100 via feedback that he / she is about to cross, has reached or has crossed the boundary of the designated first area 206. The feedback may be in the form of vibration (tactile feedback), which may include one or more mechanical pulses. Alternatively or additionally, the feedback may include audio feedback.

[0384] like Fig.17As shown, in some embodiments, the first area 206 may have a boundary 280, and the processing unit 204 may be configured to operate the feedback component when the finger of the user 50 is crossing, reaching, or moving to a position within a predetermined distance 282 from the boundary 280. In one implementation, the first designated area 206 has a square or rectangular shape with four boundaries 280. In such a case, the processing unit 204 may be configured to operate the feedback component when the finger of the user 50 crosses, reaches, or moves to a position within a prescribed distance from any one of the four boundaries 280 of the first designated area 206. The feedback component may include any or any combination of tactile feedback components, speakers, other types of components capable of generating feedback signals (e.g., visual signals displayed by the image display device 101). Therefore, as used in this specification, the term "feedback component" may include one or more components for providing one or more sensory feedback. Similarly, as used in this specification, the term "feedback" may include one or more different types of feedback.

[0385] As another example, the first area 206 may have one or more boundaries 280 that at least partially surround the reference position. For example, the first area 206 may have a square or rectangular shape surrounding the reference position (e.g., the center), or may have a circular shape surrounding the reference position (e.g., the center). In such a case, the processing unit 204 may be configured to operate a feedback component in the device 200 in response to the finger of the user 50 reaching a specified distance from the reference position.

[0386] Furthermore, in some embodiments, the processing unit 204 may be configured to operate the feedback component in the device 200 to generate different types of feedback based on different corresponding spatial relationships between one or more fingers of the user 50 relative to the first area 206. This allows the user 50 to know the extent to which his / her fingers reach one or more boundaries 280 of the designated first area 206. In one example, the different corresponding spatial relationships may be that one of the one or more fingers, or a point between two fingers, reaches a different corresponding distance 282 from the boundary 280 of the designated first area 206. In another example, the different corresponding spatial relationships may be different distances (exceeding a threshold) between (1) a point between one of the one or more fingers of the user, or two fingers, and (2) a reference position located within the designated first area 206.

[0387] In one example, the different types of feedback may include a first tactile pulse having a first amplitude and a second tactile pulse having a second amplitude different from the first amplitude. In such a case, when the user's finger moves closer to the boundary 280 of the designated first area 206 (or moves away from a reference position surrounded by one or more boundaries 280), the amplitude of the tactile pulse will increase.

[0388] In another example, the different types of feedback include a first number of tactile pulses, and a second number of tactile pulses that is different from the first number. In such a case, when the user's finger moves closer to a boundary 280 of the designated first area 206 (or moves away from a reference position surrounded by one or more boundaries 280), the number of tactile pulses will increase.

[0389] In another example, the different types of feedback include a first series of tactile pulses having a first frequency, and a second series of tactile pulses having a second frequency different from the first frequency. In such a case, when the user's finger moves closer to the boundary 280 of the designated first area 206 (or moves away from a reference position surrounded by one or more boundaries 280), the frequency of the tactile pulses will increase.

[0390] In another example, the different types of feedback may include a first audio signal having a first pitch and a second audio signal having a second pitch.

[0391] In yet another example, the different types of feedback may include a first number of audio signals and a second number of audio signals, wherein the first number is different from the second number.

[0392] In another example, the different types of feedback may include a first audio message and a second audio message different from the first audio message.

[0393] In other embodiments, the processing unit 204 may operate the feedback component to generate different types of feedback based on the user's finger reaching different boundaries 280 of the designated first area 206. For example, the designated first area 206 may have a first boundary 280 (e.g., a left boundary or a right boundary) and a second boundary 280 (e.g., a top boundary or a bottom boundary). In such a case, the processing unit 204 may be configured to operate the feedback component to generate a first type of feedback when one or more fingers of the user cross, reach, or move to a position within a specified distance from the first boundary; and may be configured to operate the feedback component to generate a second type of feedback when one or more fingers of the user cross, reach, or move to a position within a specified distance from the second boundary. Such a feature may be desirable because it allows the user 50 to know which boundary 280 his / her finger is approaching without looking at the device 200.

[0394] In addition, in other embodiments, the processing unit 204 can be configured to operate the feedback component based on the sliding direction of the user's finger. For example, assume that the designated first area 206 has a rectangular shape with a top boundary, a bottom boundary, a left boundary, and a left boundary. If the user 50 slides his / her finger upward, the boundary 280 that may be reached will be the top boundary. In such a case, the processing unit 204 can then operate the feedback component to generate a feedback type to notify the user 50 that the user's finger has reached the top boundary.

[0395] In some embodiments, it may be desirable to ensure that feedback is provided only when a touch to a location outside (or close to) the border 280 of the first area 206 is caused by the user 50 using the designated first area 206. For example, if the user 50 touches a location outside the designated first area 206, and the touch action is not part of a continuous sliding action that started from within the first area 206, the processing unit 204 may not operate the feedback component to provide any feedback. On the other hand, if the touch at a location outside the designated first area 206 (or outside of the designated first area 206 within a specified distance) is part of a sliding action that started from within the designated first area 206, the processing unit 204 may operate the feedback component to provide feedback.

[0396] Fig.18 An algorithm or method 300 for providing feedback that takes into account the above two scenarios is shown. First, the processing unit 204 determines whether there is an input signal indicating that a touch input of the user 50 is received (item 302). The input signal can be generated due to the user 50 touching any position in the touch-sensitive area of ​​the device 200. If there is no such input signal, the processing unit 204 then clears the mark (item 304). In the illustrated embodiment, the mark is used to track whether the user 50 has initiated a finger action by touching a position within the specified first area 206. If the user has initiated such an action, the mark will be set. If not, the mark will be cleared. After clearing the mark, the method loops back to item 302 so that the processing unit 204 continues to determine whether the user 50 has touched any part of the touch-sensitive area of ​​the device 200 (e.g., an area outside and inside the specified first area 206).

[0397] If the processing unit 204 determines that there is an input signal indicating that a touch input by the user 50 is received, the processing unit 204 then determines whether the touch location is located within the designated first area 206 of the screen 202 (item 306). If the touch location is located within the designated first area 206, the processing unit 204 sets the flag to indicate that a finger action involving the user 50 touching a location in the designated first area 206 has occurred (item 308). The processing unit 204 also adds the touch input data to the buffer (item 310) and uses the input data stored in the buffer to perform gesture recognition (item 312). As discussed, the buffer is configured to store input data generated by the user 50 performing finger actions on the designated first area 206 and / or other designated areas. In some embodiments, in item 312, the processing unit 204 performs gesture recognition using the input data to identify a corresponding command (finger action) for the gesture performed. The processing unit 204 also generates or identifies a corresponding command for the detected finger action performed on one or more designated areas of the screen 202. In some embodiments, the processing unit 204 is configured to send commands to the image display device 101 (eg, to the processing module 130). The processing module 130 then determines what content to display to the user 50 for viewing based on the received commands provided by the apparatus 200.

[0398] Return to Fig.18 On the other hand, if the processing unit 204 determines in item 306 that the touched location is outside the designated first area 206, the processing unit 204 then determines whether a flag was previously set (item 316). If the flag was not previously set, it means that the currently touched location is not part of a finger action initiated from a location within the designated first area 206 (e.g., the currently touched location may have been caused by the user 50 tapping, grabbing a location outside the designated first area 206, etc.). In such a case, the processing unit 204 does not operate the feedback component, and no feedback is provided to the user 50. The method then loops back to item 302, so that the processing unit 204 can continue to determine whether the user 50 has touched any part of the touch-sensitive area of ​​the device 200 (e.g., the area outside and inside the designated first area 206).

[0399] On the other hand, if the processing unit 204 determines in item 316 that the flag was previously set, this means that the currently touched location (which is outside the designated first area 206 or within a specified distance 282 from the boundary 280 of the designated first area 206) is part of the trajectory of the finger action initiated from within the first area 206. In such a case, the processing unit 204 then operates the feedback component to provide feedback (item 318). As long as feedback has been provided, the processing unit can refresh the buffer that stores input data generated from the user 50 touching the designated first area 206 (item 314). The refreshing of the buffer means that the previous input data caused by the user 50 touching the designated first area 206 is deleted, and no command is recognized and generated because the user's finger has reached or exceeded the boundary 280 of the designated first area 206.

[0400] Changes in the content of the input signal and / or sensor signal generated by the finger

[0401] As described above, the designated first area 206 in the screen 202 (which functions like a touchpad) and other designated areas (such as areas 240, 250, 260) allow the user 50 of the video display device 101 to provide various finger-generated inputs to the video display device 101. The processing unit 204 of the device 200 is configured to generate an electronic signal (caused by the finger-generated input) for causing a change in the content displayed by the image display device 101. In some cases, the electronic signal may represent an identification of the finger action. Alternatively or additionally, the electronic signal may represent the processing unit 204 based on the use of the designated areas 206, 240, 250, 260 (such as those described with reference to Figures 10-16, and Fig.18 The apparatus 200 may be configured to transmit the electronic signal to the image display device 101 via a wired connection. In other embodiments, the apparatus 200 may use a wireless communication module to convert the electronic signal into a wireless form for wireless transmission to the image display device 101. The processing module 130 of the video display device 101 receives the signal from the apparatus 200 and changes the content for display by the display subsystem 110 based on the signal.

[0402] By way of non-limiting example, the change in the content may include a change in the size of the content, a change in the position of the content, a change in the shape of the content, a change in the color of the content, a replacement of information in the content, an increase or decrease in the amount of information in the content, or any combination of the foregoing. In some cases, the change in the content caused by the input generated by the finger (generated using the specified first area 206) may be the movement of a cursor or pointer. In other cases, the change in the content caused by the input generated by the finger may be a change from a selection of an item displayed by the image display device 101 to another selection of another item. In other cases, the change in the content caused by the input generated by the finger may be the movement of an object (e.g., a photo, a computer-generated image, a cartoon, etc.) across the display of the image display device 101, wherein the movement may occur within the viewing plane of the user 50 (e.g., in the XY plane), or along the viewing depth of the user 50 (e.g., along the Z axis). In addition, in some cases, the change in the size of the content may be a change in the size of an object (e.g., a photo, a computer-generated image, a cartoon, etc.) displayed by the image display device 101.

[0403] It should be noted that the finger motion detected by processing unit 204 can be used by processing unit 204 to determine various different commands (examples of electronic signals), and processing unit 204 can send such commands to image display device 101 (wirelessly or through a wired connection) to allow image display device 101 to change the content being displayed based on the command. By way of non-limiting example, from a detected touch movement (such as Fig.11A The command determined by the image display device 101 can be processed by the image display device 101 to align elements, control the movement of a cursor, navigate within the plane of the viewing screen of the image display device 101, etc.; from a detected upward or downward slide (such as Fig. 10A The command determined by the image display device 101 can be processed by the image display device 101 to scroll the content in simple grid and list layouts; from a detected left or right swipe (such as Fig. 10B The command determined by the image display device 101 can be processed by the image display device 101 to move the next page, different part, etc. to the current viewing frame for display by the image display device 101; from two finger touch and scroll action (such as Fig. 11B The commands determined by the image display device 101 may be processed by the image display device 101 to provide inertial scrolling of the content displayed by the image display device 101 (i.e., if the two-finger touch is moved slowly, the page scrolls slowly and precisely, and if the two-finger touch is moved with a quick swipe, the page flashes); from a pinch action (such as Fig. 12A as shown) or expansion action (as Fig. 12BThe command determined by the image display device 101 can be processed by the image display device 101 to change the size of the object displayed by the image display device 101; the long hold (such as Fig.14 The command determined by the image display device 101 can be processed by the image display device 101 to open a menu, open a browser, open an application, etc., for display by the image display device 101; the tap action detected in the specified first area 206 (such as Fig.15 ) or from a tap action on a Trigger button (as Fig. 16B The commands determined by the processing unit 204 (as shown) can be processed by the image display device 101 to open an additional function menu for display by the image display device 101; etc. The above functions caused by the commands generated by the processing unit 204 are only examples. In other embodiments, the same detected finger motions described above can be used to generate other different commands for performing other functions different from the described examples.

[0404] In addition, in some embodiments, a command sequence determined from a sequence of detected finger actions may be sent from the processing unit 204 to the image display device 101 to allow the user 50 to perform various tasks on the content displayed by the image display device 101. For example, a command sequence may be provided by the processing unit 204 for 3D placement of objects and content extraction. In one implementation, the user 50 holds the "trigger button" for a long time to grab an object displayed by the image display device 101, then performs a touch movement using the first designated area 206 to move the grabbed object, and then releases the "trigger button" to place the object in the desired position. In some embodiments, the object is grabbed as long as the "trigger" button is pressed, and the object is dropped as long as the "trigger" button is released. In such a case, the user 50 may use one hand to press the "trigger" button, and the other hand is used to move the grabbed object. Alternatively, after grabbing the object, the user may release the "trigger" button, and the object will still remain grabbed. In such a case, the user may use the same hand to move the grabbed object. After placing the grabbed object as desired, the user can then tap or long hold the "trigger" button to release the grabbed object. The 3D placement feature can allow the user 50 to move any object displayed by the image display device 101. For example, the user 50 can use the above feature to drag and drop images in applications for composing slides, moving web page sliders, etc. In some embodiments, when an object is grabbed, the user 50 can use the designated first area 206 and / or head gestures to move the grabbed object, use the designated first area 206 to rotate the grabbed object (using a gesture similar to the 3D placement feature). Fig.13 ), and can use the designated first area 206 to scale the grabbed object (using something like Fig. 12A / 12B).

[0405] As another example, the processing unit 204 may detect a certain sliding action of the user 50, and in response, provide a text box for allowing the user 50 to enter text (e.g., English letters, letters or characters in other languages, numbers, punctuation marks, special characters, symbols, emoticons, text graphics, etc.). This sliding type feature may provide the user 50 with a quick and convenient way to enter text and other information.

[0406] In some embodiments, in addition to using the input signal generated by the finger action performed on the designated area 206, the processing unit 204 of the device 200 can also obtain a sensor signal associated with the orientation of the device 200, and use the combination of the input signal and the sensor signal to cause a change in the content displayed by the image display device 101. Specifically, the processing unit 204 can determine a command based on the combination of the input signal and the sensor signal, and can send such a command to the image display device 101. The image display device 101 then processes the command and changes the content based on the command. Alternatively, the processing unit 204 can determine a first command for the input signal and a second command for the sensor signal, and can send the two commands to the image display device 101. The image display device 101 then processes the command and changes the content based on the two commands.

[0407] For example, in some embodiments, the device 200 also includes an orientation sensor for sensing the orientation of the device 200. The orientation sensor can be any of an inertial measurement unit (IMU) or other types of orientation sensors. In such cases, the processing unit 204 can be configured to generate an electronic signal based on both (1) the input signal generated by the finger and (2) the sensed orientation of the device to cause the content displayed by the image display device 101 to change.

[0408] In some cases, the input signal generated by the finger may be associated with a contraction or expansion motion of the user 50. In such a case, the device processing unit 204 may be configured to generate an electronic signal based on (1) the input signal associated with the contraction or expansion motion and (2) the sensed orientation of the device 200 to cause the content displayed by the image display device 101 to change. For example, the processing unit 204 may be configured to: (1) when the device 200 is in a first orientation, if a contraction or expansion motion is sensed by the device 200, generate an electronic signal to cause the content to shrink and expand in a first plane; and (2) when the device 200 is in a second orientation different from the first orientation, if a contraction or expansion motion is sensed by the device 200, generate an electronic signal to cause the content to shrink and expand in a second plane, the second plane being different from the first plane. In some embodiments, when the main axis of the device 200 forms an angle of less than 45°, or more preferably less than 30° with the horizontal plane, the device 200 may be considered to have a first orientation. Furthermore, when the major axis of the device 200 forms an angle with the vertical plane that is less than 45°, or more preferably less than 30°, the device 200 may be considered to have a second orientation.

[0409] In addition, the first plane in which the content is reduced or expanded due to the device being in the first orientation can be perpendicular to the second plane in which the content is reduced or expanded due to the device being in the second orientation. For example, the first plane can be the YZ plane in the virtual three-dimensional environment, and the second plane includes the XY plane in the virtual three-dimensional environment.

[0410] In other embodiments, the input signal may be generated by other types of finger-generated motions instead of the contraction or expansion motion of the user 50. For example, the input signal may be caused by a sliding motion performed by the user 50. In such a case, the processing unit 204 may be configured to generate an electronic signal based on (1) the input signal associated with the sliding motion and (2) the sensed orientation of the apparatus 200 to cause the content displayed by the image display device 101 to change.

[0411] For example, the processing unit 204 may be configured to: (1) generate an electronic signal to move content in a first plane if a contraction or expansion motion is sensed by the device 200 when the device 200 is in a first orientation; and (2) generate an electronic signal to move content in a second plane if a contraction or expansion motion is sensed by the device 200 when the device 200 is in a second orientation different from the first orientation, the second plane is different from the first plane. In some embodiments, when the main axis of the device 200 forms an angle of less than 45°, or more preferably less than 30° with the horizontal plane, the device 200 may be considered to have the first orientation. In addition, when the main axis of the device 200 forms an angle of less than 45°, or more preferably less than 30° with the vertical plane, the device 200 may be considered to have the second orientation.

[0412] In addition, the first plane in which the content moves due to the device being in the first orientation can be perpendicular to the second plane in which the content moves due to the device being in the second orientation. For example, the first plane can be the YZ plane in the virtual three-dimensional environment, and the second plane includes the XY plane in the virtual three-dimensional environment.

[0413] In some embodiments, the content is in a virtual three-dimensional environment, and the processing unit 204 is configured to generate an electronic signal when a sliding motion is sensed by the apparatus 200 when the orientation of the apparatus 200 is approximately parallel to a horizontal plane (e.g., forming an angle of ±30° from the horizontal plane) to change the content displayed by the image display device 101 by moving it closer to or farther away from the user 50. The processing unit 204 is also configured to generate an electronic signal when a sliding motion is sensed by the apparatus 200 when the orientation of the apparatus 200 is approximately perpendicular to the horizontal plane (e.g., forming an angle of ±30° from the vertical plane) to move the content displayed by the image display device 101 in a vertical plane in the three-dimensional environment.

[0414] In some embodiments, processing unit 204 is configured to generate an electronic signal to expand the content in one or more directions based on the sensed orientation of device 200. For example, if device 200 is in a first orientation (e.g., an upright orientation), the electronic signal may be a command to expand the content in a first direction corresponding to the first orientation of device 200. If device 200 is in a second orientation (e.g., a face-up orientation), the electronic signal may be a command to expand the content in a second direction corresponding to the second orientation of device 200.

[0415] Furthermore, in some embodiments, processing unit 204 is configured to generate electronic signals to cause content to move or rotate based on the sensed orientation of device 200 .

[0416] Furthermore, in some embodiments, the apparatus 200 further comprises a motion sensor for sensing movement of the apparatus 200, wherein the processing unit 204 is configured to generate an electronic signal based on the sensed movement of the apparatus 200 to cause the content displayed by the image display device 101 to change (e.g., move). The motion sensor may include an accelerometer, or any other component that can detect movement and / or movement direction.

[0417] Fig.19A and 19B Examples of some of the above features are shown. Specifically, Fig.19A The device is shown in the face-up orientation. Figure 6 Examples of finger actions performed on a device, and examples of corresponding effects on displayed content. Fig.19B The device is shown in the upright orientation. Figure 6 Examples of finger actions performed on a device and corresponding effects on displayed content. Fig.19A As shown, when the device is held by the user 50 in a face-up orientation, the user 50 can perform (1) a two-finger radial movement 400a on the first area 206 designated in the screen 202 of the device 200 to rotate the object 402 displayed by the image display device 101 about the Y axis, (2) a vertical sliding movement 400b on the first area 206 designated in the screen 202 of the device 200 to move the object 402 away along the Z axis, and (3) an expansion movement 400c on the first area 206 designated in the screen 202 of the device 200 to increase the size of the object 402 along the Z axis. On the other hand, as shown in FIG. Fig.19B As shown, when the device is held by the user 50 in an upright orientation, the user 50 can perform (1) a two-finger radial movement 410a on the first area 206 designated in the screen 202 of the device 200 to rotate the object 402 displayed by the image display device 101 around the Z axis, (2) a vertical sliding movement 410b on the first area 206 designated in the screen 202 of the device 200 to move the object 402 upward (or downward) along the Y axis, and (3) an expansion movement 410c on the first area 206 designated in the screen 202 of the device 200 to increase the size of the object 402 along the Y axis. Therefore, depending on the orientation of the device 200, the same finger action will achieve different effects on the content displayed by the image display device 101.

[0418] In some embodiments, the processing unit 204 may be configured to detect six different scenarios of finger motion input and / or orientation sensor signals, and generate corresponding control signals to move the object displayed by the image display device 101 with six different degrees of freedom, respectively. For example, based on the finger motion input and / or orientation sensor signals, the object displayed by the image display device 101 may be translated along the X axis, translated along the Y axis, translated along the Z axis, rotated around the X axis, rotated around the Y axis, and rotated around the Z axis.

[0419] In some embodiments, the electronic signal generated by the processing unit 204 of the device 200 can also cause the image display device 101 to display a touch prompt so that the user 50 watching the screen of the image display device 101 can see what finger action is being detected by the processing unit 204. For example, if the user 50 is performing a two-finger touch movement on the first designated area 206 of the screen 202 to inertially move an object displayed by the image display device 101, the electronic signal generated by the processing unit 204 based on the detection of the two-finger touch movement can be sent to the image display device 101, and the image display device 101 then displays a graphic (touch prompt) to notify the user 50 that he / she is performing a two-finger touch movement. The same feature can be applied to other types of detected finger actions. Therefore, the image display device 101 can display different finger action indicators for notifying the user 50 that different corresponding finger actions have been detected by the device 200.

[0420] It should be noted that in one or more embodiments, any features described herein may be performed by the processing unit 204 of the apparatus 200 and / or the processing unit 130 of the image display device 101. Therefore, one or more features described herein as being performed by the processing unit 204 of the apparatus 200 may alternatively be performed by the processing unit 130 of the image display device 101, or by a combination of the processing unit 204 of the apparatus 200 and the processing unit 130 of the image display device 101. Similarly, one or more features described herein as being performed by the processing unit 130 of the image display device 101 may alternatively be performed by the processing unit 204 of the apparatus 200, or by a combination of the processing unit 204 of the apparatus 200 and the processing unit 130 of the image display device 101.

[0421] Gesture switching

[0422] In some embodiments, the image display device 101 may include a camera for detecting a gesture of the hand of the user 50, and the processing unit 130 of the image display device 101 may interpret the detected gesture to generate a corresponding control signal for operating the image display device 101. In some embodiments, the gesture performed by the user 50 on the designated first area 206 may be combined with the gesture detected by the camera of the image display device 101 to achieve the desired operation. For example, in some embodiments, the device 200 may display content on its screen, and the user 50 may perform a contraction action on the screen of the device 200 to extract the displayed content. The processing unit 204 detects the contraction action performed using the device 200 and interprets it as a control signal to extract the content. Then, the user 50 may perform an expansion action in the field of view of the camera of the image display device 101. The image display device 101 detects the expansion action and interprets it as a control signal to place the content at a certain position presented by the screen of the image display device 101 (or visible through it). Thus, gesture detection performed by apparatus 200 may be combined with gesture detection performed by image display device 101 to operate on content displayed by apparatus 200 and / or content displayed by image display device 101 .

[0423] In another example, the image display device 101 may display content on its screen, and the user 50 may perform a pinching motion in front of the camera of the image display device 101 to extract the displayed content. The processing unit 130 of the image display device 101 detects the pinching motion captured by the camera and interprets it as a control signal to extract the content. Then, the user 50 may perform an expansion motion on the device 200. The device 200 detects the expansion motion and interprets it as a control signal to place the content at a certain position presented by the screen of the device 200.

[0424] Furthermore, in some embodiments, content displayed by the device 200 may be moved to the "environment" of the image display device 101 by selecting the content at the device 200 and moving the content (e.g., by touch movement, sliding, etc.) to the edge of the display area of ​​the device 200. When the processing unit 204 detects that the content is moved to the edge of the display, the processing unit 204 then determines that the content is to be moved to the display screen of the image display device 101. The device 200 then sends a control signal to cause the content to be displayed by the image display device 101. After the content has been "moved" from the screen of the device 200 to the screen of the image display device 101, the user 50 may then perform additional operations on the content by using the device 200 (e.g., operating on the designated first area 206 to move the content, resize the content, etc.) and / or using hand gestures for detection by the camera of the image display device 101.

[0425] In some embodiments, the "movement" of the content from the screen of the device 200 to the screen of the image display device 101 (or vice versa) can be performed to provide time continuity. For example, as soon as the content disappears from the screen of the device 200, the processing unit 130 of the image display device 101 can immediately generate and provide an image of the content for display on the screen of the image display device 101. In other embodiments, the "movement" of the content from the screen of the device 200 to the screen of the image display device 101 (or vice versa) can be performed without the need for time continuity. For example, after the content disappears from the screen of the device 200, there may be a time lag for providing an image of the content for display on the screen of the image display device 101.

[0426] In addition, in some embodiments, the "movement" of content from the screen of the device 200 to the screen of the image display device 101 (or vice versa) can be performed to provide spatial continuity. For example, if the content on the screen of the device 200 is moved to the right edge of the display for moving the content to the screen of the image display device 101, the image display device 101 can immediately display the content next to the left edge of the screen of the image display device 101 as soon as the content is moved off the screen of the device 200. Similarly, as another example, if the content on the screen of the image display device 101 is moved to the right edge of the display for moving the content to the screen of the device 200, the device 200 can immediately display the content next to the left edge of the screen of the device 200 as soon as the content is moved off the screen of the image display device 101.

[0427] In other embodiments, the "movement" of content from the screen of the apparatus 200 to the screen of the image display device 101 (or vice versa) may be performed without requiring spatial continuity. For example, after the content on the screen of the apparatus 200 moves off the screen, the content may always appear at a predetermined position (e.g., center) on the screen of the image display device 101.

[0428] In one example of use, the above features can be implemented to allow user 50 to handle cards in a poker game. For example, user 50 can use device 200 to perform a sliding action to deal out playing cards. The playing cards can move out of the screen of device 200 and can appear on the screen of image display device 101 (or on the screen of another image display device being used by another user, where the other image display device can communicate with image display device 101 via a network such as the Internet, Wi-Fi, etc.).

[0429] Furthermore, in some embodiments, the camera of the image display device 101 for detecting user gestures may be used to view the user's hand while the user 50 is performing finger motion input on the apparatus 200. In such a case, the processing unit 130 of the image display device 101 may detect the finger motion of the user 50 and verify the detection of the finger gesture detected by the apparatus 200.

[0430] In addition, in some embodiments, the image display device 101 is configured to detect whether the user 50 is using the device 200. If the user 50 is using the device 200 to enter input, the image display device 101 can disable its gesture detection camera. Alternatively, in some examples, if the user 50 is using the device 200 to enter input, the image display device 101 can keep its camera enabled and simply avoid performing one or more gesture detection processes on the image captured by the camera. As long as the user 50 lifts his / her finger off the screen of the device 200, the image display device 101 can activate the gesture detection camera to allow the camera to take over the gesture detection function. In one embodiment, the processing unit 130 of the image display device 101 is configured to disable the gesture detection camera of the image display device 101 as long as the device 200 is receiving input generated by the finger action of the user 50. The processing unit 130 can be configured to receive a signal from the device 200 as long as the user 50 lifts his / her finger off the screen of the device 200. In response to such a signal, the processing unit 130 then activates a gesture detection camera of the image display device 101 to allow gestures detected by the camera to be used as input to the image display device 101 .

[0431] Method performed by a processing unit and / or an application in the processing unit

[0432] Fig. 20 500 according to some embodiments is shown. The method 500 may be performed by the processing unit 204 and / or an application in the processing unit 204 of the apparatus 200. The method 500 includes: designating a first area 206 in the screen 202 of the apparatus 200 to sense finger movements of a user 50 of the image display device 101, wherein the image display device 101 is configured to be worn by the user, and the apparatus 200 is different from the image display device 101 (item 502). The image display device 101 may be Figure 1-4 The method 500 further includes generating an electronic signal based on the user's finger motion sensed by the designated first area 206 of the screen 202 of the apparatus 101 to cause a change in the content displayed by the image display device 101 (item 504).

[0433] In some embodiments, the electronic signal may be a command determined by the processing unit 204 or by an application in the processing unit 204 based on gesture recognition. The electronic signal may be sent by the apparatus 200 to the processing unit 130 of the image display device 101, which then changes the content based on the electronic signal. Alternatively, if the apparatus 200 is involved in providing content for display by the image display device 101, the processing unit 204 may change the content being displayed based on the electronic signal it provides. Therefore, the electronic signal may directly or indirectly cause a change in the content displayed by the image display device 101.

[0434] Optionally, in the method 500 , the screen 202 has a touch-sensitive area 210 , and wherein the designated first area 206 is smaller than a total area of ​​the touch-sensitive area 210 .

[0435] Optionally, the method 500 may also include ignoring input generated by the user 50 using a portion of the touch-sensitive area 210 that is not part of the designated first area 206 and is not part of the designated button.

[0436] Optionally, the method 500 may further include generating a control signal in response to a finger motion of the user 50 to operate a feedback component.

[0437] Optionally, in method 500 , the first area has a boundary, and wherein the finger action of user 50 includes the user's finger crossing, reaching or moving to a position within a specified distance from the boundary.

[0438] Optionally, in method 500 , first region 206 has one or more boundaries at least partially surrounding the reference location, and wherein the control signal is used to operate the feedback component in response to a finger of user 50 reaching a specified distance from the reference location.

[0439] Optionally, the method 500 further includes generating different control signals to operate the feedback component to generate different corresponding types of feedback based on different corresponding spatial relationships between one or more fingers of the user 50 relative to the first area 206 .

[0440] Optionally, in method 500 , the different types of feedback include a first tactile pulse having a first amplitude, and a second tactile pulse having a second amplitude different from the first amplitude.

[0441] Optionally, in method 500 , the different types of feedback include a first number of tactile pulses, and a second number of tactile pulses different from the first number.

[0442] Optionally, in method 500 , the different types of feedback include a first series of tactile pulses having a first frequency, and a second series of tactile pulses having a second frequency different from the first frequency.

[0443] Optionally, in method 500 , the different corresponding spatial relationships include different distances between (1) a finger among one or more fingers of the user, or a point between two fingers, and (2) a reference position within the designated first area 206 .

[0444] Optionally, in the method 500 , the reference position includes a center of the designated first area 206 .

[0445] Optionally, in method 500 , the different distances exceed a threshold.

[0446] Optionally, in the method 500 , the different corresponding spatial relationships include one of the one or more fingers, or a point between two fingers reaching different corresponding distances from the boundary of the designated first area 206 .

[0447] Optionally, in method 500, the designated first area 206 has a first boundary and a second boundary; wherein the different types of feedback include at least a first type of feedback and a second type of feedback; wherein, when one or more fingers of the user 50 cross, reach or move to a position within a specified distance from the first boundary, the feedback component is operated to generate the first type of feedback; and wherein when one or more fingers of the user 50 cross, reach or move to a position within a specified distance from the second boundary, the feedback component is operated to generate the second type of feedback.

[0448] Optionally, in the method 500 , the first boundary includes a left boundary or a right boundary of the designated first region 206 , and the second boundary includes a top boundary or a bottom boundary of the designated first region 206 .

[0449] Optionally, in method 500, the control signal for operating the feedback component is based on the sliding direction.

[0450] Optionally, the method 500 further includes receiving an input signal associated with a contraction or expansion action performed by the user 50 on the designated first area 210 of the screen 202 .

[0451] Optionally, in the method 500 , the electronic signal is used to respond to an input signal associated with a contraction or expansion action so that the size of the content displayed by the image display device 101 changes.

[0452] Optionally, the method 500 further comprises obtaining an orientation of the apparatus 200 from an orientation sensor, wherein the electronic signal is used to cause content displayed by the image display device 101 to change based on the input signal associated with the contraction or expansion motion and the orientation of the apparatus 200 .

[0453] Optionally, in method 500, when the device 200 is in a first orientation, if a contraction or expansion motion is sensed by the designated first area 206, the content is changed by shrinking and expanding the content in a first plane; and wherein, when the device 200 is in a second orientation, if a contraction or expansion motion is sensed by the designated first area 206, the content is changed by shrinking and expanding the content in a second plane, the second plane being different from the first plane.

[0454] Optionally, in method 500, the device 200 has a first orientation when a major axis of the device 200 forms an angle less than 45° with a horizontal plane.

[0455] Optionally, in method 500, the device 200 has the second orientation when a major axis of the device 200 forms an angle of less than 45° with a vertical plane.

[0456] Optionally, in method 500 , the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0457] Optionally, in method 500 , the first plane and the second plane are relative to the virtual three-dimensional environment.

[0458] Optionally, in method 500 , the first plane is perpendicular to the second plane.

[0459] Optionally, the method 500 further includes receiving an input signal associated with a sliding action performed by the user 50 on the designated first area 206 in the screen 202 .

[0460] Optionally, in method 500 , the electronic signal is used to move content displayed by the image display device 101 in response to the sensed sliding motion.

[0461] Optionally, the method 500 further comprises obtaining an orientation of the apparatus 200 from an orientation sensor, wherein the electronic signal is used to cause content displayed by the image display device 101 to change based on the input signal associated with the sliding action and the orientation of the apparatus 200 .

[0462] Optionally, in method 500, when the device 200 is in a first orientation, if a sliding motion is sensed by the designated first area 206, the content is changed by moving the content in a first plane; and wherein, when the device 200 is in a second orientation different from the first orientation, if a sliding motion is sensed by the designated first area 206, the content is changed by moving the content in a second plane, the second plane being different from the first plane.

[0463] Optionally, in method 500, the device 200 has a first orientation when a major axis of the device 200 forms an angle less than 45° with a horizontal plane.

[0464] Optionally, in method 500, the device 200 has the second orientation when a major axis of the device 200 forms an angle of less than 45° with a vertical plane.

[0465] Optionally, in method 500 , the first plane includes a YZ plane in the virtual three-dimensional environment, and the second plane includes an XY plane in the virtual three-dimensional environment.

[0466] Optionally, in method 500 , the first plane and the second plane are relative to the virtual three-dimensional environment.

[0467] Optionally, in method 500 , the first plane is perpendicular to the second plane.

[0468] Optionally, in method 500, the content is in a virtual three-dimensional environment, and wherein the control signal is used to move the content displayed by the image display device 101 closer to or farther away from the user when a sliding motion is sensed by the designated first area 206 when the apparatus 200 is oriented approximately parallel to a horizontal plane.

[0469] Optionally, in method 500, the content is in a virtual three-dimensional environment, and wherein the control signal is used to cause the content displayed by the image display device 101 to move in a vertical plane in the three-dimensional environment when a sliding motion is sensed by the designated first area 206 when the apparatus 200 is oriented substantially perpendicular to the horizontal plane.

[0470] Optionally, the method 500 further comprises obtaining a sensor input indicating a sensed orientation of the apparatus 200 , wherein the electronic signal is used to cause content displayed by the image display device 101 to change based on the sensor input indicating the sensed orientation of the apparatus 200 .

[0471] Optionally, in method 500 , control signals are used to cause the content to change by causing the content to expand in one or more directions based on sensor input indicating a sensed orientation of device 200 .

[0472] Optionally, in method 500 , the control signal is used to cause the content to change by rotating the content based on a sensor input indicating a sensed orientation of device 200 .

[0473] Optionally, in method 500 , the control signal is used to cause the content to change by causing the content to move based on a sensor input indicating a sensed orientation of device 200 .

[0474] Optionally, the method 500 further comprises obtaining a sensor input indicating the sensed movement of the apparatus 200 , wherein the control signal is used to cause the content displayed by the image display device 101 to change based on the sensor input indicating the sensed movement of the apparatus 200 .

[0475] Optionally, in method 500 , electronic signals are used to cause the content to change by causing the content to move based on sensor input indicating sensed movement of device 200 .

[0476] Optionally, in method 500, device 200 is a handheld device.

[0477] Optionally, the handheld device comprises a cellular phone, a smart phone, a personal digital assistant (PDA), or a tablet computer.

[0478] Optionally, in the method 500 , when the designated first area 206 of the screen 202 is sensing the finger motion of the user 50 , the designated first area 206 of the screen 202 has no displayed object.

[0479] Optionally, the method 500 further includes operating the screen 202 to display a grid of points in the designated first area 206 of the screen 202 .

[0480] Optionally, the method 500 further includes changing a characteristic of one or more of the points in response to the user 50 touching a portion of the one or more of the displayed points in the designated first area 206 of the screen 202 .

[0481] Optionally, the method 500 further includes designating a second area of ​​the screen 202 as a first button.

[0482] Optionally, in method 500, the first button is a "home" button.

[0483] Optionally, the method 500 further includes designating a third area of ​​the screen 202 as a second button.

[0484] Optionally, in method 500, the second button is a "Toggle" button.

[0485] Optionally, the method 500 further includes designating a fourth area of ​​the screen 202 as a third button.

[0486] Optionally, in method 500, the third button is a "buffer" button.

[0487] Optionally, the method 500 further includes: designating a second area of ​​the screen 202 as a keyboard activation button, and in response to the user 50 touching the designated second area of ​​the screen, operating the screen 202 to display the keyboard.

[0488] Optionally, the method 500 further includes communicating with the image display device 101 wirelessly.

[0489] Optionally, the method 500 further includes communicating with the image display device 101 via a cable.

[0490] Optionally, in the method 500 , the apparatus 200 includes a non-transitory medium storing instructions, and wherein the action of designating the first area 206 of the screen 202 for sensing the finger action of the user 50 is performed based on the instructions.

[0491] Optionally, in method 500, the change in the content includes a change in the size of the content, a change in the position of the content, a change in the shape of the content, a change in the color of the content, a replacement of information in the content, an increase or decrease in the amount of information in the content, or any combination of the foregoing.

[0492] In some embodiments, method 500 may also include: detecting gestures or finger movements, such as one finger touch movement, two finger touch movement, one finger slide, two finger slide, two finger contraction, two finger expansion, two finger radial movement, tap, double-click, or any combination of the foregoing; determining a command for the corresponding detected gesture or finger movement; and sending the command to the image display device.

[0493] In addition, one or more embodiments, any features (eg, functions, items, steps, etc.) in the method 500 may be executed by the processing unit 204 of the apparatus 200 and / or the processing unit 130 of the image display device 101 .

[0494] Dedicated processing system

[0495] In some embodiments, the method 500 described herein may be performed by a processing unit 204 executing an application or by an application. The application may include a set of instructions. In one implementation, a dedicated processing system may be provided having a non-transitory medium storing a set of instructions for the application. Execution of the instructions by the processing unit 204 of the device 200 will cause the processing unit 204 to perform the features described herein. Fig.21As shown, in some embodiments, the dedicated processing system may include one or more servers 600. A non-transitory medium storing instructions 610 for an application may be implemented in the server 600, which is configured to provide a set of instructions 610 for downloading to the device 200 over a network (such as over the Internet). The server 600 may be configured to obtain a request from the device 200 to download the set of instructions 610, process the request, and provide the set of instructions 610 based on the request and / or the processing of the request.

[0496] In some embodiments, processing of the request by the server 600 may include: authenticating the device 200, authenticating the user 50 of the device 200, determining the brand of the device 200, determining the model of the device 200, or any combination of the foregoing.

[0497] It should be noted that server 600 is a special purpose processing system in that it contains instructions that are executed by the device's processing unit to provide unique tangible effects in the real world.The features provided by server 600 provide improvements in the art of image display devices and systems as described herein.

[0498] By way of non-limiting example, the instructions 610 provided by the server 600 may include instructions for: designating a first area 206 of the screen 202 of the device 200 for sensing finger motion; designating a second area 240 of the screen 202 of the device 200 as a first button; designating a third area 250 of the screen 202 as a second button; designating a fourth area 260 of the screen 202 as a third button; designating another area 270 of the screen 202 as a keyboard activation button; generating an electronic signal based on a user's finger motion sensed by the designated first area of ​​the screen of the device to cause a change in content displayed by the image display device; ignoring a finger motion sensed by the user using a portion of the screen other than the designated first area; The method comprises the steps of: generating an input generated by a part of the touch-sensitive area that is not part of the designated button and is not part of the designated button; generating a control signal to operate the feedback component in response to a user's finger movement; operating the feedback component in response to the user's finger reaching a specified distance from a reference position; generating different control signals to operate the feedback component based on different corresponding spatial relationships between one or more fingers of the user relative to the first area to generate different corresponding types of feedback; operating the feedback component to generate a first type of feedback when one or more fingers of the user crosses, reaches or moves to a position within a specified distance from a first boundary; operating the feedback component to generate a first type of feedback when one or more fingers of the user crosses, reaches or moves to a position within a specified distance from a second boundary. The device comprises a feedback component to generate a second type of feedback; operating the feedback component based on the sliding direction; generating an electronic signal in response to an input signal associated with a contraction or expansion action to change the size of the content displayed by the image display device; obtaining the orientation of the device from an orientation sensor; generating an electronic signal based on the input signal associated with the contraction or expansion action and the orientation of the device to change the content displayed by the image display device; generating an electronic signal in response to the sensed sliding action to move the content displayed by the image display device; generating an electronic signal based on the input signal associated with the sliding action and the orientation of the device to change the content displayed by the image display device; when the device is in the first orientation, if the second orientation is specified by the second orientation sensor, the electronic signal is generated. When a sliding motion is sensed by the designated first area, an electronic signal is generated to move the content in a first plane; when the device is in a second orientation different from the first orientation, if a sliding motion is sensed by the designated first area, an electronic signal is generated to move the content in a second plane, the second plane being different from the first plane; when the orientation of the device is approximately parallel to a horizontal plane, when a sliding motion is sensed by the designated first area, an electronic signal is generated to move the content displayed by the image display device closer to or farther away from the user; when the orientation of the device is approximately perpendicular to the horizontal plane, when a sliding motion is sensed by the designated first area, an electronic signal is generated to move the content displayed by the image display device in a vertical plane in a three-dimensional environment;Generate an electronic signal based on a sensor input of a sensed orientation of a pointing device to cause content displayed by an image display device to change; Generate an electronic signal based on a sensor input of a sensed orientation of a pointing device to cause content to change by causing content to expand in one or more directions; Generate an electronic signal based on a sensor input of a sensed orientation of a pointing device to cause content to change by causing content to rotate; Generate an electronic signal based on a sensor input of a sensed orientation of a pointing device to cause content to change by causing content to move; Obtain a sensor input of a sensed movement of a pointing device; Generate an electronic signal based on a sensor input of a sensed movement of a pointing device to cause content displayed by an image display device to change; Generate an electronic signal based on a sensor input of a sensed movement of a pointing device to cause content to change by causing content to move; When a designated first area of ​​the screen is sensing a user's finger action, turn off pixels in the designated first area of ​​the screen so that the designated first area has no displayed objects; Operate the screen of the device to display a grid of points in the designated first area of ​​the screen; In response to a user touching a portion of one or more of the displayed points in the designated first area of ​​the screen, change characteristics of one or more of the points, or any combination of any of the foregoing.;

[0499] In addition, the instructions 610 provided by the server 600 may include instructions for: detecting gestures or finger movements, such as one finger touch movement, two finger touch movement, one finger slide, two finger slide, two finger contraction, two finger expansion, two finger radial movement, tap, double-click, or any combination of the foregoing; determining a command for the corresponding detected gesture or finger movement; and sending the command to the image display device.

[0500] In other embodiments, the apparatus 200 may also be considered a special purpose processing system. Specifically, the apparatus 200 is a special purpose processing system because it contains instructions stored in its non-transitory medium for execution by the processing unit 204 to provide unique tangible effects in the real world. The features provided by the apparatus 200 (as a result of the execution of the instructions by the processing unit 204) provide improvements in the art of image display devices and systems, as described herein.

[0501] Fig. 22 1 is a block diagram illustrating an embodiment of a dedicated processing system 1600 that can be used to implement the various features described herein. For example, in some embodiments, the processing system 1600 can be used to implement one or more servers 600. In addition, in other embodiments, the processing system 1600 can be used to implement the device 200.

[0502] The processing system 1600 includes a bus 1602 or other communication mechanism for communicating information and a processor 1604 coupled to the bus 1602 for processing information. The processor system 1600 also includes a main memory 1606, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1602 for storing information and instructions to be executed by the processor 1604. The main memory 1606 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 1604. The processor system 1600 also includes a read-only memory (ROM) 1608 or other static storage device coupled to the bus 1602 for storing static information and instructions for the processor 1604. A data storage device 1610, such as a magnetic disk, solid state disk, or optical disk, is provided and coupled to the bus 1602 for storing information and instructions.

[0503] The processor system 1600 may be coupled to a display 1612, such as a screen, via the bus 1602 for displaying information to a user. In some cases, if the processing system 1600 is part of an apparatus that includes a touch screen, the display 1612 may be a touch screen. An input device 1614 including alphanumeric and other keys is coupled to the bus 1602 for communicating information and command selections to the processor 1604. Another type of user input device is a cursor control 1616, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1604 and for controlling cursor movement on the display 1612. The input device typically has two degrees of freedom on two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane. In some cases, if the processing system 1600 is part of an apparatus that includes a touch screen, the input device 1614 and the cursor control may be a touch screen.

[0504] In some embodiments, the processor system 1600 may be used to perform various functions described herein. According to some embodiments, such use is provided by the processor system 1600 in response to the processor 1604 executing one or more sequences of one or more instructions contained in the main memory 1606. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read from another processor-readable medium (e.g., storage device 1610) into the main memory 1606. The execution of the sequence of instructions contained in the main memory 1606 causes the processor 1604 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequence of instructions contained in the main memory 1606. In alternative embodiments, hard-wired circuits may be used instead of software instructions or in combination with software instructions to implement the various embodiments described herein. Therefore, the embodiments are not limited to any specific combination of hardware circuits and software.

[0505] The term "processor-readable medium" as used herein refers to any medium that participates in providing instructions to processor 1604 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, solid-state disks, or magnetic disks, such as storage device 1610. Non-volatile media can be considered as examples of non-transitory media. Volatile media include dynamic memory, such as main memory 1606. Volatile media can be considered as examples of non-transitory media. Transmission media include coaxial cables, copper wiring, and optical fibers, including the wiring that comprises bus 1602. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0506] Common forms of processor-readable media include, for example, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, any other physical medium with a pattern of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, a solid state disk or any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a processor can read.

[0507] Various forms of processor-readable media may be involved in carrying one or more sequences of one or more instructions to the processor 1604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a network (such as the Internet). The processing system 1600 may receive data over a network line. The bus 1602 carries the data to the main memory 1606, from which the processor 1604 retrieves and executes the instructions. The instructions received by the main memory 1606 may optionally be stored on the storage device 1610 before or after execution by the processor 1604.

[0508] Processing system 1600 also includes a communication interface 1618 coupled to bus 1602. Communication interface 1618 provides bidirectional data communication coupled to network link 1620, which is connected to local network 1622. For example, communication interface 1618 can be a local area network (LAN) card that provides a data communication connection to a compatible local area network (LAN). Wireless connections can also be implemented. In any such implementation, communication interface 1618 sends and receives electrical, electromagnetic or optical signals that carry data streams representing various types of information.

[0509] The network link 1620 typically provides data communication to other devices through one or more networks. For example, the network link 1620 can provide a connection to a host computer 1624 or device 1626 through a local network 1622. The data stream sent through the network link 1620 may include electrical, electromagnetic, or optical signals. The signals through the various networks and the signals on the network link 1620 and the signals through the communication interface 1618 (which carry data to and from the processing system 1600) are exemplary forms of carrier waves that transmit information. The processing system 1600 can send messages and receive data including program code through the network, network link 1620, and communication interface 1618.

[0510] It should be noted that the device 200 described herein is not limited to having the described functions and features, and the device 200 may be configured to provide other features for use with the image display device 101. As mentioned above, in some examples, some or all of such functions and features may be provided at least in part by means of an application (also referred to herein as an "application") running on the device 200. By way of non-limiting example, the apparatus 200 may be configured to allow the user 50 to: register an account associated with use of the image display device 101; log into an application using credentials established on an account; retrieve or reset credentials for an account; switch between user profiles; automatically synchronize user account data between devices; manage an account associated with use of the image display device 101; manage the configuration and use of the image display device 101; obtain supplemental learning and support features for the image display device 101; obtain help with troubleshooting the image display device 101 and / or apparatus 200; access privacy, security, and data policies associated with an account; access an app store to obtain applications for the image display device 101 and / or apparatus 200; perform a search query for an application; view an application detail page; view an order or purchase history; receive push notifications (e.g., application updates, etc.); perform media extraction from a local device; open a media file (e.g., photos, videos, etc.); select individual media files to send to the image display device 101, select individual intermediate files to be sent from the image display device 101 to the apparatus 200; receive notifications about the use of the image display device 101 (e.g., notifications from applications of the image display device 101); receive notifications about the use of the image display device 101 from third-party developers; manage smartphone notification settings for different applications of the image display device 101; find the image display device 101 or other image display devices; obtain the battery status of the image display device 101; mirror the image display device 101 on the apparatus 200 (wherein, the content displayed on the screen of the image display device 101 and / or the environment around the user 50 as viewed by the screen of the image display device 101 and captured by the camera of the image display device 101 can be sent to the apparatus 200 for display by the apparatus 200); or any combination of the above. The processing unit 204 of the apparatus 200 can be configured to execute instructions to cause the apparatus 200 to provide one or more of the above features.

[0511] Exemplary aspects of the present disclosure and details about material selection and manufacturing have been described above. For other details of the present disclosure, these details can be understood in conjunction with the above patents and disclosures and in a manner generally known or understood by those skilled in the art. This is also true for the method-based aspects of the present disclosure in terms of additional actions that are usually or logically employed.

[0512] In addition, although the present disclosure has been described with reference to several examples optionally incorporating various features, the present disclosure is not limited to those described or indicated with respect to each variation of the present disclosure. Various changes may be made to the present disclosure described without departing from the spirit and scope of the present disclosure, and equivalents (whether referenced herein or not included for the sake of brevity) may be substituted. In addition, where a range of values ​​is provided, it is understood that each intermediate value between the upper and lower limits of the range, and any other stated or intermediate values ​​within the stated range, are included in the present disclosure.

[0513] Moreover, it is contemplated that any optional feature of the disclosed variations described may be set forth or claimed independently, or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that there are multiple identical items. More specifically, as used herein and in the claims associated herewith, the singular forms "a," "an," "said," and "the" include plural referents unless specifically stated otherwise. It should also be noted that any claim may be drafted to exclude any optional element. Therefore, this statement is intended to serve as antecedent basis for the use of exclusive terminology such as "solely," "only," or the use of a "negative" limitation in conjunction with the recitation of claim elements.

[0514] The breadth of the present disclosure is not limited to the examples provided and / or subject specification, but is limited only by the scope of the claim language associated with the present disclosure.

[0515] In the foregoing description, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the above process flow is described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A device comprising: Touch-sensitive display; a feedback component configured to provide non-visual feedback; as well as one or more processors operatively coupled to the touch-sensitive display and the feedback assembly and communicatively coupled to a wearable device, the one or more processors being configured to: receiving data from the wearable device indicating whether the apparatus is within a field of view of a camera of the wearable device; as well as Based on data indicating whether the device is within the field of view of the camera of the wearable device, switch between a first mode and a second mode, wherein in the first mode, the one or more processors are configured to communicate one or more portions of a user interface to a user through the feedback component, wherein, in the second mode, the one or more processors are configured to present the user interface on the touch-sensitive display of the device without utilizing the feedback component of the device, wherein the one or more processors are configured to: switch to the second mode if the data indicates that the device is within the field of view of the camera of the wearable device, and switch to the first mode if the data indicates that the device is outside the field of view of the camera of the wearable device.

2. The device according to claim 1, wherein: The feedback component is a tactile actuator.

3. The device according to claim 1, wherein: The feedback component is a speaker.

4. The device according to claim 1, wherein: The user interface is a graphical user interface for controlling one or more functions of the wearable device.

5. The device according to claim 4, wherein: The graphical user interface includes a plurality of graphical control elements.

6. The device according to claim 5, wherein: When in the mode, the one or more processors are configured to suppress presentation of one or more of the plurality of graphical control elements on the touch-sensitive display.

7. The device according to claim 5, wherein: When in the first mode, the one or more processors are configured to present a limited version of the graphical user interface on the touch-sensitive display.

8. The device according to claim 7, wherein: The limited version of the graphical user interface on the touch-sensitive display is a version in which one or more of the plurality of graphical control elements are not shown, a version in which a brightness level of one or more of the plurality of graphical control elements is reduced, or a combination thereof.

9. The device according to claim 5, wherein: When in the first mode, the one or more processors are configured to communicate one or more locations on the touch-sensitive display at which one or more of the plurality of graphical control elements are otherwise presented in the second mode.

10. A processor-implemented method comprising: receiving an indication from a wearable device whether an apparatus having a touch-sensitive display is within a field of view of a camera of the wearable device; as well as Based on the data, switch between a first mode and a second mode, wherein, in the first mode, one or more portions of a user interface are communicated to a user of the wearable device via a feedback component of the device, and wherein, in the second mode, the user interface is presented on the touch-sensitive display of the device without using the feedback component of the device, wherein if the data indicates that the device is within the field of view of the camera of the wearable device, switch to the second mode; if the data indicates that the device is outside the field of view of the camera of the wearable device, switch to the first mode.

Images