VIRTUAL INTERFACE FOR CONTROLLING AN IoT DEVICE

Abstract

A virtual interface application presented in Augmented Reality (AR) for controlling Internet of Things (IoT) products is described. The virtual interface application enables a user of the portable electronic device to activate a virtual control interface overlay on the display, receive a selection from the user using her hand or foot, and send a control signal to a nearby IoT product that performs an action according to the selection. The virtual control interface is presented on the display as a virtual object in the AR environment. The virtual interface application includes a foot tracking tool for detecting an intersection between a foot location (in a physical environment) and a virtual surface location (in an AR environment). When the intersection is detected, the virtual interface application sends a control signal with an instruction to the IoT product.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 901,630, filed on September 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of augmented reality (AR) experiences for electronic devices, including portable and wearable devices such as eyewear. More particularly, but not by way of limitation, this disclosure describes an AR application for displaying a virtual control interface that enables a user to interact with and control an IoT device. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital glasses, headsets, headbands, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays. Users sometimes refer to information on these devices during physical activities such as exercise.

[0005] The so-called "Internet of Things" (IoT) refers to and includes the following physical products that are embedded with sensors, software, and other technologies for connecting and exchanging data with other devices in a network (usually via the Internet). For example, IoT products are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided to act as smart home hubs to connect different smart home products. IoT devices are also used in many other applications. Application layer protocols and supporting frameworks have been provided to implement such IoT applications. For example, some IoT products include an application programming interface (API) that allows IoT products to pair with other products and electronic devices (such as portable computers) and communicate with them separately. Artificial intelligence has also been combined with the IoT infrastructure to achieve more efficient IoT network operation, improve human-computer interaction, and enhance data management and analysis.

[0006] Virtual reality (VR) technology generates a complete virtual environment including realistic images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move around in the virtual environment and interact with virtual objects. AR is a VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the impression that the virtual objects are actually present in the environment, especially when the virtual objects appear and behave like real objects. Cross-reality (XR) is generally understood as an umbrella term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar element, with additional uppercase or lowercase letters referring to the specific element.

[0008] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:

[0009] Figure 1A is a side view of an eyewear device suitable for use with an example virtual control system (right);

[0010] Figure 1B yes Figure 1A a perspective partial cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0011] Figure 1C yes Figure 1A Side view of the eyewear device (left);

[0012] Figure 1D yes Figure 1C a cross-sectional perspective view of the optical components and electronics in a portion of the eyewear device shown;

[0013] Figure 2A and 2B is a rear view of an eyewear device used in an example virtual control system;

[0014] Figure 3 is a block diagram illustrating an example of using the example eyewear device shown in any of the preceding figures to capture visible light;

[0015] Figure 4is a functional block diagram of an example virtual control system including an eyewear device and a server system connected via various networks;

[0016] Figure 5 Is configured for Figure 4 An illustration of an example hardware configuration of a mobile device of an example virtual control system;

[0017] Figure 6 is a stereogram of a user in an example environment, including physical and virtual objects, for reference in describing an example process known as simultaneous localization and mapping (SLAM);

[0018] Figure 7 is a flow chart listing steps in an example method of presenting a virtual control interface on a display;

[0019] Fig. 8A is an illustration of an example virtual control interface presented on a display of a portable electronic device;

[0020] Figure 8B Is the user Fig. 8A An illustration of example interactions between virtual control interfaces resulting in corresponding IoT actions;

[0021] Fig. 9 is an illustration of an example interaction between a user and an example virtual control interface presented on a display of an electronic eyewear device. DETAILED DESCRIPTION

[0022] A virtual control system for use with AR applications on portable electronic devices (including mobile phones and wearable devices such as electronic eyewear devices). The virtual control system enables a user of the portable electronic device to view a virtual control interface presented on a display near an IoT product. The user can use the body (e.g., hands, feet) to make selections or otherwise interact with the virtual control interface to adjust one or more controllable features of the IoT product (e.g., on, off, volume, brightness).

[0023] Various embodiments and details are described with reference to an example for presenting a virtual control interface in an augmented reality environment to control an IoT product using a portable electronic device with a camera and a display. In an example embodiment, a method involves capturing a video data frame with a camera and detecting an IoT product at an IoT product location in a physical environment using the captured video data frame. The example method also includes determining a portable electronic device position relative to the IoT product location, and then presenting a virtual control interface on a display based on the IoT product location and the portable electronic device location. The example virtual control interface includes a virtual surface that is virtually presented at a surface location in the environment relative to the display. The example method also includes detecting a foot shape that performs a foot action at a foot location relative to the physical environment, and then detecting an intersection between the foot location and the surface location. The intersection and the foot action (e.g., tap, slide) correspond to an IoT action (e.g., actuating a switch, adjusting another feature). If an intersection is detected, the example method includes sending a control signal to the IoT product, wherein the control signal includes instructions for use by the IoT product to perform an IoT action.

[0024] Although various systems and methods are described herein with reference to foot motions near the base of an IoT product, the techniques described herein can be applied to detect and track any part of the body (e.g., hands, fingers, arms, and legs) relative to substantially any type of control interface on any surface. For example, one or more hand motions can be detected near a virtual control interface of an IoT product located on a tabletop or other surface, or mounted on a wall. The virtual control interface described herein can be used in a variety of IoT consumer products, such as lights, speakers, and fans, as well as IoT products for a wide variety of commercial, medical, and industrial applications.

[0025] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0026] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0027] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to other parts of the item may be described as "distal." For example, the end of an item closest to the object may be called the proximal end, while the generally opposite end may be called the distal end.

[0028] For purposes of illustration and discussion, the orientation of the eyeglass device, associated components, and any complete device incorporating an eye scanner and camera, such as those shown in any of the accompanying drawings, is given by way of example only. In operation for a particular variable optical processing application, the eyeglass device may be oriented in any other orientation (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eyeglass device. Additionally, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and are not limiting with respect to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.

[0029] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time recognition and tracking of body parts, such as hands and fingers, arms and legs, and feet, is one of the most challenging and processing-intensive tasks in the field of computer vision.

[0030] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, instrumentalities and combinations particularly pointed out in the appended claims.

[0031] In a sample configuration, an eyewear device with augmented reality (AR) capabilities is used in the system described herein. Eyewear devices are expected to be used in the system described herein because such devices are scalable and customizable to achieve a personalized experience, achieve the effect of being applied anytime, anywhere, and ensure user privacy by only allowing the user to see the transmitted information. In a sample configuration, the eyewear device can be used without any dedicated hardware, such as the SPECTACLES available from Snap, Inc. of Santa Monica, California. TM .

[0032] like Figures 1A-1D As shown, the eyewear device 100 includes a first camera 114A and a second camera 114B. The cameras 114 capture image information of a scene from separate viewpoints. The captured images can be used to project a three-dimensional display onto an image display for three-dimensional (3D) viewing.

[0033] Cameras 114 are sensitive to wavelengths in the visible light range. Each of cameras 114 defines a different forward-facing field of view that overlaps to enable generation of a three-dimensional depth image, e.g., first camera 114A defines a first field of view 111A, and second camera 114B defines a second field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular location and orientation in space. Fields of view 111 have overlapping fields of view 304 ( Figure 3 ). When the camera captures an image, objects or object features outside the field of view 111 are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or degree to which the image sensor of the camera 114 picks up electromagnetic radiation of a given scene in the captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0034] In an example configuration, one or both cameras 114 have a field of view of 100° and a resolution of 480x480 pixels. The "coverage angle" describes the range of angles that the lens of the camera 114 can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the camera's film or sensor, which may include some vignetting (e.g., the image becomes darker toward the edges when compared to the center). If the coverage angle of the camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle.

[0035] Examples of suitable cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of 480p (e.g., 640 x 480 pixels), 720p, 1080p, or higher resolutions. Other examples include cameras 114 that can capture high-definition (HD) video at high frame rates (e.g., 30 to 60 frames per second or more) and store recordings at a resolution of 1216 by 1216 pixels (or higher).

[0036] The eyewear device 100 can capture image sensor data and geographic location data from the camera 114, which is digitized by the image processor to be stored in the memory. The camera 114 captures corresponding raw images (e.g., left and right raw images) in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0037] To capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4 ) may be coupled to the camera 114 to receive and store visual image information. The image processor 412 or another processor controls the operation of the camera 114 to act as a stereo camera that simulates human binocular vision, and a time stamp may be added to each image. The time stamp on each pair of images allows the images to be displayed together as part of a 3D projection. 3D projection creates an immersive, lifelike experience that is desirable in a variety of environments, including virtual reality (VR) and video games.

[0038] Figure 1B yes Figure 1A 1 is a perspective cross-sectional view of the right corner 110A of the eyewear device 100 depicting the first camera 114A, additional optical components and electronics. Figure 1C yes Figure 1A 1 is a side view (left) of an example hardware configuration of the eyewear device 100 showing the second camera 114B of the camera system. Figure 1D yes Figure 1C 1 is a perspective cross-sectional view of a left corner 110B of the eyewear device 100 depicting a second camera 114B, additional optical components, and electronics of the camera system.

[0039] like Figure 1BAs shown in the example of FIG. 1 , the eyeglass device 100 includes a first camera 114A and a circuit board 140A, which may be a flexible printed circuit board (PCB). The first hinge 126A connects the right corner 110A to the first temple 125A of the eyeglass device 100. In some examples, the first camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the first temple 125A or the first hinge 126A.

[0040] The right corner 110A includes a corner body 190 and a corner cap. Figure 1B The corner cap is omitted in the cross section of FIG. Disposed within the right corner 110A are various interconnected circuit boards 109, such as a flexible PCB 140A, which include circuit boards for the first camera 114A, (one or more) microphones 139, (one or more) speakers 191, low-power wireless circuit systems (e.g., for communicating via Bluetooth TM wireless short-range network communications), high-speed wireless circuit systems (for example, controller circuits for wireless LAN communications via Wi-Fi).

[0041] The first camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A As shown, the right edge 107A of the eyeglass frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens. The eyeglass frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the eyeglass frame 105. In this example, the first camera 114A has an outward facing field of view 111A ( Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the right corner 110A where the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0042] like Figure 1D As shown in the example of FIG. 1 , the eyeglass device 100 includes a second camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The second hinge 126B connects the left corner 110B to the second temple 125B of the eyeglass device 100. In some examples, the second camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the second temple 125B or the second hinge 126B.

[0043] The left corner 110B includes a corner body 190 and a corner cap. Figure 1D The corner cap is omitted in the cross section of FIG. Disposed inside the right corner 110A are various interconnected circuit boards (such as a flexible PCB 140B) that include controller circuitry for the second camera 114B.

[0044] The camera 114 is coupled to or disposed on a corresponding flexible PCB 140 and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A 10A and 107B are connected to the left corner 110B and include openings for the camera cover lenses. The frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the frame 105. In this example, the camera 114 has a corresponding outward facing field of view 111 ( Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the corresponding corner 110, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0045] Figure 2A and 2B An example hardware configuration of an eyewear device 100 including two different types of image displays is depicted. The eyewear device 100 is sized and shaped in a form configured to be worn by a user. The form of the eyewear is shown in the illustrated example. The eyewear device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0046] In the eyeglass example, the eyeglass device 100 includes a frame 105, which includes a right rim 107A connected to a left rim 107B via a nose bridge 106, and the nose bridge 106 is configured to receive the user's nose to support the eyeglass device 100 on the user's head. The right rim 107A includes a first rim 175A, which holds a first optical element 180A. The left rim 107B includes a second rim 175B, which holds a second optical element 180B. Figure 2BAs shown, in some embodiments, each optical element 180A, 180B includes an integrated image display (e.g., a first display 182A and a second display 182B). As used herein, the term "lens" is intended to include a transparent or translucent glass or plastic sheet having a curved or flat surface that causes light to converge or diverge, or to cause little or no convergence or divergence.

[0047] A touch-sensitive input device such as a touch pad 181 is located on the first temple 125A. As shown, the touch pad 181 can have a clearly visible border, or include a raised or otherwise tactile edge that provides feedback to the user about the location and border of the touch pad 181; alternatively, the border can be subtle and not easily seen or felt. The eyeglass device 100 can include a touch pad on the other side that operates independently or in conjunction with the touch pad 181.

[0048] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface (GUI) displayed by the eyewear device on the image display to allow the user to browse and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0049] Detecting finger input on the touchpad 181 can achieve a variety of functions. For example, touching any position on the touchpad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180. Tapping or double-clicking the touchpad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, moving to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, moving to the previous item, icon, video, image, page, or slide. The touchpad 181 can be positioned substantially anywhere on the eyeglass device 100.

[0050] In one example, a recognized single tap gesture on touch pad 181 initiates selection or pressing of a GUI element in an image presented on the image display of optical assembly 180. Adjusting the image presented on the image display of optical assembly 180 based on the recognized gesture can be a primary action that selects or submits a GUI element on the image display of optical assembly 180 for further display or execution.

[0051] Figure 2A1 is an example hardware configuration of the eyeglass device 100 in which the right corner 110A supports a microphone 139 and a speaker 191. The microphone 139 includes a transducer that converts sound into a corresponding electrical audio signal. The microphone 139 in the example shown is positioned to have an opening facing inward toward the wearer to facilitate receiving sound waves, such as human speech including verbal commands and questions. Additional or differently oriented openings may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more microphones 139 that are configured to operate together or independently and are positioned at various locations on the eyeglass device 100.

[0052] The speaker 191 includes an electroacoustic transducer that converts an electrical audio signal into a corresponding sound. The speaker 191 is controlled by one of the processors 422, 432 or by the audio processor 413 ( Figure 4 ) control. As shown, the speaker 191 in this example includes a series of rectangular apertures that face inward to direct sound to the wearer. Additional or differently oriented apertures may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more speakers 191 that are configured to operate together (e.g., in stereo, to produce surround sound in a zone) or independently and are positioned at various locations on the eyeglass device 100. For example, one or more speakers 191 may be incorporated into the frame 105, temple 125, or corner 110 of the eyeglass device 100.

[0053] Despite Figure 2A and Figure 2B 100 is shown as having two optical elements 180, but depending on the application or intended user of the eyeglass device 100, the eyeglass device 100 may include other arrangements, such as a single optical element (or it may not include any optical element 180). As further shown, the eyeglass device 100 includes a right corner 110A adjacent to the right lateral side 170A of the frame 105 and a left corner 110B adjacent to the left lateral side 170B of the frame 105. The corners 110 may be integrated into the frame 105 on the respective sides 170 (as shown), or may be implemented as separate components attached to the frame 105 on the respective sides 170. Alternatively, the corners 110A, 110B may be integrated into temples attached to the frame 105 (not shown).

[0054] In one example, each image display of optical assembly 180 includes an integrated image display (eg, first display 182A and second display 182B). Figure 2AAs shown, each optical assembly 180 has a display 182 including a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or other such display. Each optical assembly 180 also includes one or more optical layers 176, which may include any combination of lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components. Optical layer ( Figure 2A The optical layer 176A-N (shown as 176A-N) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eye. The prisms of the optical layer 176A-N extend over all or at least a portion of the corresponding lens ring 175 formed in the left and right lens edges 107 to allow the user to see the second surface of the prism when the user's eye looks through the corresponding lens edge 107. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105, and the display matrix 177 covers the prism so that photons and light emitted by the display matrix 177 strike the first surface. The prism is designed in size and shape so that light is refracted within the prism and directed toward the user's eye through the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex to direct light toward the center of the eye. The prisms may optionally be sized and shaped to magnify the image projected by the display matrix 177 , and light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 177 .

[0055] In one example, the optical layers 176A-N may include an LCD layer that is transparent (holding the lens open) unless and until a voltage is applied that renders the layer opaque (closing or blocking the lens). The image processor 412 on the eyewear device 100 may be programmed to apply a voltage to the LCD layer, thereby creating an active shutter system, making the eyewear device 100 suitable for viewing visual content when displayed as a 3D projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0056] In another example, the image display device of the optical assembly 180 has a display 182 that includes a display such as Figure 2BEach optical assembly 180 includes a corresponding laser projector 150, such as a three-color laser projector using a scanning mirror or a galvanometer. Each laser projector 150 is disposed in or on a corresponding temple 125 of the eyeglass device 100. In this example, each optical assembly 180 includes one or more optical strips (shown as Figure 2B 155A-N in FIG. 155A-N), these optical strips span the width of the lens of each optical component 180 or span the depth of the lens between the front and back surfaces of the lens and are spaced apart.

[0057] As photons projected by the laser projector 150 travel through the lens of each optical assembly 180, the photons encounter optical strips 155A-N. When a particular photon encounters a particular optical strip, the photon is redirected toward the user's eye, or it passes to the next optical strip. The combination of modulation of the laser projector 150 and modulation of the optical strips controls specific photons or light beams. In an example, the processor controls the optical strips 155A-N by issuing a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical assemblies 180, the eyeglass device 100 may include other arrangements, such as a single or three optical assemblies, or each optical assembly 180 may have a different arrangement depending on the application or intended user of the eyeglass device 100.

[0058] Figure 3 3D scene 306, a first raw image 302A captured using a first camera 114A, and a second raw image 302B captured using a second camera 114B. As shown, the first field of view 111A can overlap with the second field of view 111B. The overlapping field of view 304 represents the portion of the image captured using the two cameras 114. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302 can be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.

[0059] To capture stereo images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a 3D scene 306 are captured at a given moment—a first raw image 302A captured using a first camera 114A and a second raw image 302B captured using a second camera 114B. When the pair of raw images 302 are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on an optical assembly 180 of the eyewear device, on another display (e.g., an image display 580 on a mobile device 401), or on a screen.

[0060] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0061] Figure 4 4 is a functional block diagram of an example virtual control system 400, which includes an eyeglass device 100, a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). As shown, the virtual control system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the eyeglass device 100 and the mobile device 401.

[0062] As described herein, the eyewear device 100 includes one or more cameras 114 that capture still images, video images, or both still images and video images. The camera 114 may have direct memory access (DMA) to the high-speed circuit system 430 and be used as a stereo camera. The camera 114 may be used to capture initial depth images that may be rendered as three-dimensional (3D) models that are texture-mapped images of red, green, and blue (RGB) imaging scenes. The device 100 may also include a depth sensor that uses infrared signals to estimate the position of an object relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters and one or more infrared cameras 410.

[0063] The eyeglass device 100 also includes two image displays (one associated with the right side 170A and one associated with the left side 170B) of the optical assembly 180. The eyeglass device 100 also includes an image display driver 442, an image processor 412, a low power circuit system 420, and a high speed circuit system 430. The image display of the optical assembly 180 is used to present images, including still images, video images, or both still images and video images. The image display driver 442 is coupled to the image display of the optical assembly 180 to control the display of the image.

[0064] Figure 4The components of the eyeglass device 100 shown in FIG. 1 are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the rim or temple. Alternatively or additionally, the depicted components may be located in the corners, frame, hinge, or bridge of the eyeglass device 100. The camera 114 includes a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible light or light capturing element that can be used to capture data, including still images or video of a scene with an unknown object.

[0065] like Figure 4 As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In the example, an image display driver 442 is coupled to the high-speed circuit system 430 and is operated by the high-speed processor 432 to drive the image display of the optical assembly 180. The high-speed processor 432 can be substantially any processor capable of managing high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436.

[0066] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eyeglass device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 also executes a software architecture for the eyeglass device 100, which is used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuit system 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to as Wi-Fi herein. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 436.

[0067] The low power circuit system 420 includes a low power processor 422 and a low power wireless circuit system 424. The low power wireless circuit system 424 and the high speed wireless circuit system 436 of the eyeglass device 100 may include a short range transceiver (Bluetooth TM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). Mobile device 401 (including a transceiver that communicates via low power wireless connection 425 and high speed wireless connection 437) can be implemented using details of the architecture of eyewear device 100, as can other elements of network 495.

[0068] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the cameras 114A, 114B, one or more infrared cameras 410, the image processor 412, and images generated for display by the image display driver 442 on the image display of each optical assembly 180. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, separate element of the eyewear device 100. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage addressing of the memory 434 so that the low-power processor 422 initiates the high-speed processor 432 whenever a read or write operation involving the memory 434 is to be performed.

[0069] like Figure 4 As shown, various elements of the eyeglass device 100 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both. For example, the infrared camera 410 (including an infrared transmitter in some embodiments), the user input element 491 (e.g., a button switch, a touch pad 181, a microphone 139), and the inertial measurement unit (IMU) 472 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both.

[0070] like Figure 5 As shown, CPU 540 of mobile device 401 may be coupled to camera system 570, mobile display driver 582, user input layer 591, and memory 540A, discussed in further detail below.

[0071] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eyeglass device 100 and the mobile device 401 over the network 495 .

[0072] The output component of the eyewear device 100 includes a visual element, such as Figure 2A and 2BEach lens or optical component 180 described is associated with an image display (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The eyeglass device 100 may include an indicator facing the user (e.g., an LED, a speaker 191, or a vibration actuator) or a signal facing outward (e.g., an LED, a speaker 191). The image display of each optical component 180 is driven by an image display driver 442. In some example configurations, the output components of the eyeglass device 100 also include additional indicators, such as an audible element (e.g., a speaker 191), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers 191 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outwardly facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include an LED, speaker 191, or actuator configured and positioned to be sensed by an observer.

[0073] The user input elements 491 of the eyewear device 100 may include alphanumeric input components (e.g., a touch screen or touchpad 181 configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured components), pointer-based input components (e.g., a mouse, touchpad 181, trackball, joystick, motion sensor, or other directional instrument), tactile input components (e.g., a button switch, a touch screen or touchpad 181 or other tactile configured element that senses the location, force, or location and force of a touch or touch gesture), and audio input components (e.g., a microphone 139), and the like. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0074] In some examples, the eyeglass device 100 includes a collection of motion sensing components referred to as an IMU 472. The motion sensing component may be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope senses the angular velocity of the device 100 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a GPS unit 473), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0075] The IMU 472 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the device 100. For example, acceleration data collected from the accelerometer can be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in linear coordinates x, y, and z). Angular velocity data from the gyroscope can be integrated to obtain the position of the device 100 (in spherical coordinates). The programming for calculating these useful values ​​can be stored in the memory 434 and executed by the high-speed processor 432 of the eyewear device 100.

[0076] The eyeglass device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the eyeglass device 100. For example, the peripheral device element may include any I / O component including an output component, a motion component, a positioning component, or any other such component described herein. For example, the biometric sensor may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identifying people (e.g., recognition based on voice, retina, facial features, fingerprints, or electrical biosignals (such as electroencephalogram data)), etc.

[0077] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eyewear device 100 using both a low power wireless connection 425 and a high speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0078] like Figure 4 As shown, the virtual control system 400 includes a computing device, such as a portable electronic device 401 (e.g., a mobile device), coupled to the eyewear device 100 via a network 495. The virtual control system 400 includes a memory (e.g., a non-transitory computer-readable medium) for storing instructions and a processor for executing instructions. In some embodiments, the memory and processing functions of the virtual control system 400 can be shared or distributed between the processors and memories of the eyewear device 100, the portable electronic device 401 (e.g., a mobile device), and / or the server system 498.

[0079] In some implementations, virtual control system 400 includes one or more elements or modules, referred to herein as virtual interface application 910 , foot tracking tool 912 , positioning system 915 , image processing system 920 , and speech recognition module 925 .

[0080] In some implementations, virtual interface application 910 renders and presents virtual control interface 800 on a display as described herein.

[0081] In some embodiments, the foot tracking tool 912 involves comparing pixel-level data in one or more captured video data frames with a set of foot shapes and foot motions (e.g., postures) (which can be stored in a database or library). The detected foot shape data may include three-dimensional coordinates of ankles, tarsal bones, intertarsal joints, and other bone or soft tissue landmarks found in the captured frames. These data are compared with the foot shape and foot motion data stored in the database or library until the best match is found. In some examples, the process includes calculating the sum of geodesic distances between the detected foot shape coordinates and a set of intertarsal coordinates of each foot shape stored in the database or library. The sum within a configurable threshold accuracy value represents a match. In another example embodiment, the process of foot tracking involves using a machine learning algorithm to compare pixel-level data about foot shapes in one or more captured video data frames with a set of images including foot shapes and foot motions.

[0082] In some embodiments, the positioning system 915 obtains positioning data for determining the current position of the portable electronic device 401 (e.g., a mobile phone) or the electronic eyewear device 100 relative to the physical environment 600. For example, the positioning system 915 can access the video data frame 900 captured using the camera 114B to determine the current eyewear device position 840 relative to the physical environment 600 in three-dimensional coordinates (with or without reference to data from other sources, such as an inertial measurement unit or IMU 472). As used herein, the term "video data frame" refers to video motion data captured using one or more cameras 114A, 114B coupled to the eyewear device 100, including images, spatial data, and related information captured using substantially any sensor component of the camera in any form and at any sampling rate. In some embodiments, the positioning data can be derived from the motion data frame captured using the IMU 472, from data collected by the GPS unit 473, or from a combination thereof.

[0083] In some embodiments, the image processing system 920 cooperates with the image display driver 442 and the image processor 412 to generate and present a virtual control interface 800 as described herein on a display of a portable electronic device 401 (e.g., a mobile phone) or an electronic eyewear device 100. In some embodiments, the virtual control interface 800 includes one or more virtual surfaces 802, each of which is virtually presented at a surface location in the physical environment 600 relative to the display.

[0084] In some implementations, the speech recognition module 925 receives human speech, converts the received speech into audio data frames, recognizes queries or requests based on the audio data, and performs actions related to and in response to the recognized queries or requests.

[0085] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programming to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0086] Mobile device 401 may include camera 570, which includes at least two cameras (e.g., a first visible light camera and a second visible light camera with overlapping fields of view) or at least one camera and a depth sensor with substantially overlapping fields of view. Flash memory 540A may also include multiple images or videos generated via camera 570.

[0087] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into the screen used by image display 580 .

[0088] Examples of touch screen type mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, therefore, Figure 5 A block diagram illustration of an example mobile device 401 is provided with a user interface including a touch screen input layer 591 for receiving input (through touch, multi-touch, gestures, etc., by hand, stylus, or other implement) and a graphics display 580 for displaying content.

[0089] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as for short-range network communication (such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0090] To generate location coordinates for positioning the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 401 may utilize either or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, based on a cellular network, Wi-Fi, or Bluetooth TM The positioning system of the present invention can generate accurate position coordinates, especially when used in combination. Such position coordinates can be sent to the eyewear device via XCVR 510, 520 through one or more network connections.

[0091] In some examples, the mobile device 401 includes a collection of motion sensing components called an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the mobile device 401. The motion sensing component can be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the mobile device 401 relative to three orthogonal axes (x, y, z) (including the acceleration caused by gravity). The gyroscope senses the angular velocity of the mobile device 401 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together the position, orientation, and motion data of the device relative to six degrees of freedom (x, y, z, pitch, roll, yaw) (sometimes referred to as 6DOF). The magnetometer (if present) senses the heading of the mobile device 401 relative to magnetic north.

[0092] The IMU 572 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the device mobile device 401. For example, the acceleration data collected from the accelerometer can be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the mobile device 401 (in linear coordinates x, y, and z). The angular velocity data from the gyroscope can be integrated to obtain the position of the mobile device 401 (in spherical coordinates). The programming for calculating these useful values ​​can be stored in one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the mobile device 401.

[0093] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). For example, transceivers 510, 520 provide two-way wireless communication of information (including digitized audio signals, still images and video signals, web page information for display, and web-related input), as well as various types of mobile message communications to / from mobile device 401.

[0094] The mobile device 401 also includes a microprocessor that acts as a central processing unit (CPU); Figure 4 The CPU 540 in the computer system. A processor is a circuit having the following elements that are structured and arranged to perform one or more processing functions, typically performing various data processing functions. Although discrete logic components can be used, the example utilizes components that constitute a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that combine electronic components for performing the functions of a CPU. For example, CPU 540 can be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuit systems can be used to constitute the CPU 540 or processor hardware in smartphones, laptop computers, and tablet computers.

[0095] The CPU 540 acts as a programmable main controller for the mobile device 401 by configuring the mobile device 401 to perform various operations, for example, according to instructions or programming executable by the CPU 540. For example, such operations may include various general operations of the mobile device, as well as operations related to programming for applications on the mobile device. Although the processor can be configured by using hard-wired logic, a typical processor in the mobile device is a general-purpose processing circuit that is configured by executing programming.

[0096] The mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include flash memory 540A, random-access memory (RAM) 540B and other memory components 540C as needed. RAM 540B serves as short-term storage for instructions and data processed by CPU 540, for example, as working data processing memory. Flash memory 540A typically provides long-term storage.

[0097] Thus, in the example of mobile device 401, flash memory 540A is used to store programming or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or similar systems.

[0098] The processor 432 in the eyeglass device 100 can construct a map of the environment around the eyeglass device 100, determine the position of the eyeglass device in the map of the environment, and determine the relative positioning of the eyeglass device and one or more objects in the mapped environment. The processor 432 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to construct a map and determine the position and positioning information. The sensor data includes images received from one or two of the cameras 114A, 114B, one or more distances received from the laser rangefinder, positioning information received from the GPS unit 473, motion and acceleration data received from the IMU 572, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information. In the context of augmented reality, the SLAM algorithm is used to build and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods (such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection). In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., 30 frames per second), the SLAM algorithm updates the map and the location of objects at least as frequently as the frame rate; in other words, the map and localization are calculated and updated 30 times per second.

[0099] The sensor data includes one or more images received from one or both cameras 114A, 114B, one or more distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 472, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information.

[0100] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms (simultaneous localization and mapping) and other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, foot tracking, etc.) are depicted. A portable electronic device 401 (e.g., a mobile phone) and a user 602 of the electronic eyewear device 100 are present in the example physical environment 600 (in Figure 6, the example physical environment is an interior room). The processor 432 of the eyeglass device 100 (or the processor of the mobile phone 401) uses the captured image data to determine the position of the electronic device 100, 401 relative to one or more physical objects 604 (e.g., including the IoT object 50) within the environment 600, constructs a map of the environment 600 using a coordinate system of the environment 600 (e.g., a Cartesian coordinate system (x, y, z)), and determines the position of the device 100, 410 relative to the coordinate system. In addition, in some embodiments, the processor determines the precise orientation and position (x-prime, y-prime, z-prime, roll, pitch, and yaw) of the portable electronic device 401 or the eyeglass device 100 in the environment by using two or more position points (e.g., three position points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more position points 606 associated with two or more objects 604a, 604b, 604c. An application on the portable electronic device 401 or the eyewear device 100 can present a virtual object 608 (such as Figure 6 keys shown) to view them on the display during the augmented reality experience.

[0101] In some examples, the positioning system 915 includes a virtual marker 610a associated with a virtual object 608 in the physical environment 600. In an augmented reality environment, in some embodiments, markers are registered at locations in the physical environment 600 to assist the electronic device in the task of tracking and updating the location of users, devices, and objects (virtual and physical) relative to the physical environment. Markers are sometimes registered to high-contrast physical objects (such as relatively dark objects mounted on a light-colored wall, such as framed pictures 604a) to assist cameras and other sensors in the task of detecting markers. Markers can be assigned and registered in memory by the eyewear device 100 operating in the environment. In some embodiments, markers are assigned and registered in the memory of other devices in the network.

[0102] The positioning system 915 tracks physical objects and virtual objects within the physical environment 600 relative to the eyeglass device 100. For physical objects 604 (e.g., example floor safe 604c, example IoT object 50), the positioning system 915 continuously analyzes the captured images of the physical environment 600 to identify the object 604 and determine its position relative to the eyeglass device 100 (e.g., by applying a SLAM algorithm). The positioning system 915 maintains and updates the determined position information of the physical object 604 in a memory so that the physical object 604 is tracked as the eyeglass device 100 moves through the physical environment 600. For virtual objects 608 (e.g., keys), the positioning system 915 establishes or specifies an initial position of the virtual object 608 corresponding to a position in the environment 600 or the physical object 604 (or, in some embodiments, a position relative to the eyeglass device 100). The positioning system 915 maintains and updates position information of the virtual object 608, such as according to a movement algorithm associated with the virtual object 608, in response to the eyeglass device 100 moving through the environment, or a combination thereof, thereby tracking the virtual object 608 as the eyeglass device 100 moves through the environment.

[0103] The marker may be encoded with information or otherwise linked to information. The marker may include positioning information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eyewear device 100. The set of data includes information about the marker 610a, the positioning (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker positioning may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6 The corners of the roughly rectangular marker 610a shown in . The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eyewear device 100, or other coordinate systems. One or more virtual objects associated with the marker 610a can include any of a variety of materials (including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces, and experiences) and combinations or sequences of such materials. In this context, any type of content that can be stored in memory and retrieved when the marker 610a is encountered or associated with an assigned marker can be classified as a virtual object. For example, Figure 6 The virtual key 608 shown in FIG. 6 is a virtual object displayed as a 2D or 3D still image at a marked location.

[0104] In one example, the marker 610a can be registered in memory as being located on the physical object 604a (e.g., Figure 6 In another example, the marker may be registered in memory as a specific location relative to the eyewear device 100.

[0105] Figure 7 700 is a flowchart of an example method of controlling an IoT product 50 using a portable electronic device 401. Although the steps are described with reference to the portable electronic device 401 (e.g., a mobile phone) or the eyewear device 100 described herein, for other types of devices, those skilled in the art will understand other implementations of the steps based on the description herein. One or more of the steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in combination with additional steps. Certain steps may be omitted or repeated in certain applications.

[0106] In some embodiments, the virtual interface application 910 described herein is launched in response to receiving a selection through a user interface (e.g., selecting from a menu, pressing a button, using a touchpad) or through some other input method (e.g., a gesture detected in a captured image, a finger touch 681 on the touchpad 181, a voice command).

[0107] In some embodiments, an IoT product 50 (e.g., a light, a speaker, a fan) is paired with a portable electronic device 401 or an electronic eyeglass device 100. Some IoT products 50 include an application programming interface (API) (e.g., a product API 52) that allows the IoT product 50 to pair with and otherwise communicate with other products and electronic devices (such as the portable electronic device 401 and the electronic eyeglass device 100). In some embodiments, the product API 52 includes information such as a product type 54 (e.g., a light, a speaker, a fan), and in some embodiments, a list of one or more controllable features 56 (e.g., on, off, brightness, volume, speed) associated with the IoT product 50.

[0108] Fig. 8A is an illustration of an example virtual control interface 800 presented on the display 580 of the portable electronic device 401. In this example, the IoT product 50 is a speaker, as shown. Of course, the IoT product 50 exists and is observable in the physical environment 600. When presented on the display 580 by the virtual interface application 910 described herein, the virtual control interface 800 is observable.

[0109] In some embodiments, the virtual interface application 910 controls the process of pairing the IoT product 50 with the portable electronic device 410. In some embodiments, the pairing process uses the product API 52. In some embodiments, the IoT product 50 includes an associated product API 52, which includes a product type 54 (e.g., "speaker") and a list of one or more controllable features 56 (e.g., a list of natural sounds that can be played, such as Fig. 8A shown).

[0110] Figure 8B Is the user Fig. 8A , resulting in a corresponding IoT action 60 (e.g., playing a nature sound labeled “rain”).

[0111] Fig. 9 is an illustration of an example interaction between a user and an example virtual control interface 800 presented on the display 182b of the electronic eyeglass device 100. In some embodiments, the eyeglass device 100 includes a speech recognition module 925 as described herein and a microphone 139 coupled to the speaker 191. In some embodiments, the speech recognition module 925 configures the processor 432 to perceive human speech, convert the received speech into an audio data frame, identify a first query based on the converted audio data frame, and then respond to and perform an action based on the identified first query. For example, the human speech may include a verbal command (e.g., "show virtual control interface"), and in response, the identified first query causes the virtual interface application 910 to present the virtual control interface 800 on the display.

[0112] In some embodiments, the IoT product 50 does not include an API. The IoT product 50 is typically connected to other products in the Internet or network. Due to this connection, the communication between and among the IoT products 50 typically includes data or information indicating the product type 54 (e.g., lamp, speaker, fan). In this regard, the product type 54 is generally known, even when the IoT product 50 does not include a product API 52. In some embodiments, the virtual interface application 910 described herein includes an inference engine that is configured to infer a set of possible controllable features 56 associated with the IoT product 50 based solely on the product type 54. For example, the IoT product 50 may be sharing data indicating that the product type 54 is a "speaker". In some embodiments, the inference engine will infer a set of possible controllable features 56 (e.g., on, off, variable volume, a list of available natural sounds) because such features are typically associated with a specific product type (e.g., a speaker or sound system). For another example, when the product type 54 is "fan", in some embodiments, the inference engine will infer a set of possible controllable features 56 (e.g., an on-off toggle switch for power, variable fan speed (from minimum to maximum), a set of predefined fan speeds (low, medium, high), an on-off toggle switch for oscillation (stationary or moving), and the like). When the product type 54 is "LED light", in some embodiments, the inference engine will infer a set of possible controllable features 56 (e.g., an on-off toggle switch for power, variable brightness (from minimum to maximum), variable color selection (in a continuous cycle of hues or colors), and the like).

[0113] Figure 7 Block 702 in the example method describes example steps of capturing a video data frame 900 with a camera 570 of a portable electronic device 401. In some embodiments, the process of capturing a video data frame 900 is ongoing during active use of the portable device 401. In other examples, the capture process begins in response to receiving a selection through a user interface or through some other input means. In some embodiments, the example method steps, at block 702, include at least temporarily storing the captured video data frame 900 in a memory on the portable electronic device 401 so that the video data frame 900 can be used for purposes including processing and analysis.

[0114] Figure 7Block 704 in the example steps of detecting the IoT product 50 at the IoT product location 790 in the physical environment 600 based on the captured video data frame 900. As described herein, in some embodiments, the IoT product 50 has been paired with or otherwise wirelessly communicated with the portable electronic device 401. In this regard, when the IoT product 50 is within proximity of the portable electronic device 401, the detection process occurs.

[0115] Fig. 8A An example IoT product 50 is shown located in a physical environment 600 at an IoT product location 790 .

[0116] In some embodiments, the process of detecting the IoT product 50 includes determining or estimating the IoT product location 790 relative to the physical environment 600. In some embodiments, the current IoT product location 790 is estimated using the positioning system 915 described herein.

[0117] In some embodiments, the positioning system 915 configures the virtual interface application 910 to obtain positioning data based on the captured video data frames 900, and in some embodiments, based on motion data collected by an inertial measurement unit (IMU). In some embodiments, the positioning system 915 constructs a virtual map of one or more objects within the camera field of view 904 using a SLAM algorithm, as described herein, and the SLAM algorithm updates the map and the positions of the objects at least as frequently as the camera captures the video data 900.

[0118] In some embodiments, the process of determining or estimating the IoT product location 790 relative to the physical environment 600 includes calculating a correlation between the IoT product location 790 and the current portable electronic device location 440 (e.g., Fig. 8A ). The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the IoT product location 790 and the current portable electronic device location 440. Because the positioning process occurs frequently, the correlation process between the IoT product location 790 and the portable electronic device location 440 produces accurate and near real-time tracking of the IoT product location 790 relative to the portable electronic device location 440.

[0119] Block 706 describes example steps of using the captured video data frame 900 to determine the portable electronic device location 440 relative to the IoT product location 790. As the portable electronic device 401 moves through the physical environment 600, its location relative to the IoT product location 790 changes.

[0120] Fig. 8AExample portable electronic device 410 is shown in physical environment 600 located at portable electronic device location 440 .

[0121] In some implementations, portable electronic device location 440 is estimated using positioning system 915 as described herein.

[0122] In some embodiments, the positioning system 915 configures the virtual interface application 910 to obtain positioning data based on the captured video data frames 900, and in some embodiments, based on motion data collected by an inertial measurement unit (IMU). In some embodiments, the positioning system 915 uses a SLAM algorithm to build a virtual map of one or more objects within the camera's field of view, as described herein, and the SLAM algorithm updates the map and the position of the objects at least as frequently as the camera captures the video data 900.

[0123] In some embodiments, the process of determining or estimating the portable electronic device location 440 relative to the IoT product location 790 includes calculating a correlation between the two locations. The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the current portable electronic device location 440 and the current IoT product location 790. Because the positioning process occurs frequently, the correlation process produces accurate and near real-time tracking of the portable electronic device location 440 relative to the IoT product location 790. In some embodiments, the process of estimating the current portable electronic device location 440 is based on motion data frames captured using an IMU, or based on video data frames 900 captured using a camera, or a combination of both.

[0124] In some embodiments, determining the portable electronic device location 440 relative to the IoT product location 790 includes determining whether the IoT product location 790 is within a threshold height 51 of a reference plane 620 in the physical environment 600. In some embodiments, the reference plane 620 (e.g., Fig. 8A 6. In some embodiments, the threshold height 52 relative to the reference plane 620 is predefined and configurable. For example, 36 inches may be a threshold height 52 for an IoT product 50 that is typically supported by a floor or placed on the ground.

[0125] In related aspects, presenting the virtual control interface 800 on the display 580 (as described herein at block 708) includes, in some embodiments, presenting the virtual control interface 800 adjacent to the virtual plane 822 relative to the reference plane 620. For example, Fig. 8AAs shown, example virtual plane 822 can be located a few inches above reference plane 620 (eg, to facilitate presenting virtual control interface 800 in the foreground relative to the floor in physical environment 600 ).

[0126] Figure 7 Block 708 in the embodiment recites example steps for presenting a virtual control interface 800 on the display 580 based on the IoT product location 790 and the portable electronic device location 440. In this regard, the virtual control interface 800 is presented on the display 580 at a consistent location relative to the IoT product location 790, even when the portable electronic device 401 moves through the physical environment 600. For example, Fig. 8A As shown, the virtual control interface 800 is presented on the display 580 near the base of the IoT product 50 .

[0127] In some embodiments, the process of presenting the virtual control interface 800 includes providing one or more tools to the user to adjust or otherwise configure the virtual control interface 800, as described herein, or to selectively start and stop presentation. For example, the user can pause or stop (or restart) the process of presenting the virtual control interface 800 by speaking a voice command, pressing a button on the eyewear device 100, performing a gesture, or tapping a finger touch 681 on the touchpad 181.

[0128] In some embodiments, the virtual control interface 800 includes a virtual surface 802 virtually rendered at a surface location 812 relative to the physical environment 600, such as Fig. 8A shown.

[0129] In some embodiments, the process of presenting the virtual control interface 800 includes moving or otherwise adjusting the surface location 812 relative to the IoT product 50. In some embodiments, the virtual interface application 910 moves or otherwise adjusts the surface location 812 so that the virtual control interface 800 will be presented in the foreground relative to the IoT product 50. In some embodiments, the virtual interface application 910 automatically moves or otherwise adjusts the surface location 812 without user input. In some embodiments, the surface location 812 is configurable so that the process of presenting the virtual control interface 800 includes providing the user with one or more tools to adjust or otherwise configure the surface location 812.

[0130] In some embodiments, Fig. 8AAs shown, the virtual control interface 800 includes a plurality of virtual surfaces 802 (e.g., pie-shaped segments presented in an arc around the base of the IoT product 50). In some embodiments, each of the plurality of virtual surfaces 802 is associated with a corresponding surface location 812. In this regard, the unique surface location 812 for each virtual surface 802 facilitates detection of an intersection 820 between the foot location 84 and the surface location 812, as described herein.

[0131] In some implementations, presenting the virtual control interface 800 includes presenting the virtual surface 802 as an overlay relative to the physical environment 600, such that the virtual control interface 800 is continuously visible in the foreground relative to real objects in the physical environment 600. Fig. 8A As shown, virtual control interface 800 is presented as an overlay (eg, in the foreground, above the floor) relative to physical environment 600 (eg, the floor) such that virtual control interface 800 remains visible relative to other objects in physical environment 600 .

[0132] In some embodiments, the process of presenting the virtual control interface 800 includes rendering or otherwise graphically generating the virtual surface 802 in a predefined and configurable size and shape. Fig. 8A As shown, the virtual control interface 800 is presented at a size that nearly fills the available space on the display 580. In some embodiments, the relative size of the virtual control interface 800 automatically changes based on the IoT product position 790, the portable electronic device position 440, and the available space on the display 580.

[0133] In some embodiments, the process of presenting the virtual control interface 800 includes presenting a mark 803 on the display 580 near the virtual surface 802. In some embodiments, the mark 803 is associated with one or more of the controllable features 56 (e.g., on, off, brightness, volume, speed, sound type) of the IoT product 50. For example, Fig. 8A As shown, a tag 803 (eg, “Traffic”) is associated with one of the controllable features 56 that can be generated by the IoT product 50 (eg, traffic sounds).

[0134] Figure 7 Block 710 in FIG. 7 depicts example steps for detecting foot shape 80 using captured video data frame 900. For example, Fig. 8A Detected foot shape 80 at foot position 84 relative to physical environment 600 is shown.

[0135] In some implementations, the process of detecting the foot shape 80 at the foot position 84 includes analyzing the captured video data frame 900 using a foot tracking tool 912 as described herein. Figure 8B The example foot shape 80 in FIG. 8 is a single shoe with shoelaces. The foot tracking tool 912 as described herein detects various other foot shapes 80, including bare feet, feet in the field of view of a camera (e.g., Fig. 9 In some embodiments, the foot tracking tool 912 involves comparing pixel-level data in one or more captured video data frames with a collection of foot shapes and foot movements (e.g., postures) (which can be stored in a database or library).

[0136] In some embodiments, the process of detecting the foot shape 80 at the foot position 84 includes analyzing in the captured video data frame 900 whether the foot shape 80 is detected anywhere within the field of view of the camera 570 within a predefined and configurable minimum duration (e.g., two seconds). Similarly, in some embodiments, the process of detecting the foot position 84 includes analyzing in the captured video data frame 900 whether the foot shape 80 is at the same or nearby foot position 84 (e.g., within a threshold foot position proximity) within a predefined and configurable minimum duration (e.g., one second).

[0137] Of course, the foot position 84 may change over time as a person attempts to position the foot near the desired selection. In some embodiments, the process of detecting the foot shape 80 at the foot position 84 includes detecting the last or final position of the foot in the captured video data frame 900 before the foot is removed from the camera's field of view. In this example, the process of detecting the last or final foot position 84 includes detecting one or more captured video data frames 900 in which the foot shape 80 is no longer detected; and analyzing the previously captured video data frames 900 to determine the last or final foot position 84.

[0138] Figure 7 Block 712 in illustrative examples of steps for detecting an intersection 820 between the foot position 84 and the surface position 812 (associated with the virtual surface 802 ) using the captured frame of video data 900 .

[0139] For example, Figure 8BAn example intersection 820 between the foot position 84 and the surface position 812 is shown. The detected foot position 84 is close to, but not exactly overlapping, the surface position 812. In some embodiments, the intersection 820 is detected if the foot position 84 is detected within a predefined and configurable proximity of the surface position 812. In related aspects, the foot position 84 and the surface position 812 both include three-dimensional coordinates in some embodiments. The process of detecting the intersection 820 includes estimating the proximity (e.g., in three dimensions) between the detected foot position 84 and the established surface position 812.

[0140] Figure 8B The example virtual surface position 812 in FIG. 8 is located near the center of the semicircular virtual surface 802. In another example, Fig. 9 , an example virtual surface location 812 is located near one edge of a semicircular virtual surface 802—the semicircular virtual surface 802 includes a plurality of selectable segments (e.g., pie-shaped segments that are presented in a semicircular arc around the base of the IoT product 50). In some embodiments, each of the plurality of segments is established as a virtual surface 802 associated with its respective surface location 812. In other embodiments, a single virtual surface location 812 is established (e.g., Fig. 9 ), and using coordinates (e.g., in two or three dimensions) relative to a single virtual surface location 812 to establish any associated sub-portions or segments. In this aspect, the process of detecting intersections 820 includes estimating the proximity between the detected foot location 84 and any sub-portions or segments of the virtual surface 802.

[0141] In some embodiments, the detected foot shape 80 is performing a foot motion 82, such as a tap, a slide, or a hold. In this aspect, the foot motion 82 or movement is evaluated relative to the foot position 84. In some embodiments, the foot motion 82 includes one or more movements, such as touching the foot position 84, hovering over the foot position 84, tapping the foot position 84 and then lifting, tapping the foot position 84 and holding the foot, and sliding and stopping (or not stopping) relative to the foot position 84. The virtual interface application 910 can be configured to recognize other foot motions and their associated movements relative to the foot position 84. In some embodiments, the intersection 820 and the foot motion 82 (e.g., a tap) correspond to a specific IoT action 60 (e.g., activating an on-off switch). Process

[0142] Figure 7Block 714 in describes an example step of sending a control signal 55 to the IoT product 50 based on the detected intersection 820 and the foot motion 82. In some embodiments, the process of sending the control signal 55 includes sending the control signal 55 to the IoT product 50 using a wireless transceiver, sending the control signal 55 to a cloud-based service, which in turn transmits the control signal 55 to the IoT product 50, or otherwise broadcasting the control signal 55 using methods and means that facilitate the delivery of the control signal 55 to the IoT product 50. In some embodiments, the control signal 55 includes instructions configured for use by the IoT product 50 to facilitate the implementation or execution of the associated IoT action 60. For example, when the detected foot motion 82 (e.g., a tap) corresponds to the IoT action 60 (e.g., activating an on-off switch), the control signal 55 includes instructions for the IoT product 50 to perform the IoT action 60 (e.g., switching the on-off switch from its current position to another position).

[0143] Figure 7 Block 716 in describes example steps for the IoT product 50 to perform or otherwise execute an IoT action 60 (e.g., instructions of the control signal 55) received in the control signal 55. In some embodiments, the process of performing the IoT action 60 includes associating a foot motion 82 (e.g., sliding clockwise relative to a foot position 84) with a particular controllable feature 56 (e.g., volume), and then adjusting the feature 56 (e.g., increasing the volume) based on the foot motion 82 (e.g., sliding clockwise is associated with increasing the volume).

[0144] The virtual interface application 910 includes various foot actions 82, each corresponding to one or more IoT actions 60. In one example, a foot action 82 including tapping or pressing a foot position 84 corresponds to an IoT action 60 of selecting (e.g., activating a switch, switching a switch from one setting to another, and selecting a setting, etc.). In another example, a foot action 82 including hovering near a foot position 84 corresponds to an IoT action 60 of selecting. In another example, a foot action 82 including pressing and holding the foot near a foot position 84 corresponds to an IoT action 60 of adjusting a variable feature (e.g., volume, brightness) until the foot is lifted. In another example, a foot action 82 including sliding the foot along a portion of the virtual control interface 800 corresponds to an IoT action 60 of adjusting a variable feature (e.g., volume, brightness) until the foot stops sliding or moves away from the virtual surface position 812.

[0145] These steps are described with reference to a portable electronic device 401 (e.g., a mobile phone). In some embodiments, the methods and systems described herein can be executed and operated on the eyeglass device 100, the mobile phone 401, or another electronic device. For example, in some embodiments, the portable electronic device 401 is an electronic eyeglass device 100 as described herein. In this example, the camera 570 is the eyeglass camera 114B, the display 580 is the eyeglass display 182B, the IMU 572 is the eyeglass IMU 472, the memory elements 540A, 540B, 540C are the eyeglass memory 434, etc.

[0146] Any functions of the eyewear device 100, mobile device 401, and server system 498 described herein may be embodied in one or more computer software applications or programming instruction sets, as described herein. According to some examples, a "function," "multiple functions," "application," "multiple applications," "instruction," "multiple instructions," or "programming" is a program (or programs) that executes the functions defined in the program. Various programming languages ​​may be used to generate one or more applications that are structured in a variety of ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , Mobile software running on a mobile phone or other mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article.

[0147] Thus, machine-readable media can take the form of tangible / non-temporary storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer device or the like, such as can be used to implement the client device, media gateway, transcoder, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common computer-readable media forms include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions / program codes to a processor for execution.

[0148] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.

[0149] It will be understood that the terms and expressions used herein have the common meanings given by such terms and expressions relative to their corresponding corresponding explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second, etc.) can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only includes those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0150] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range that is consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, a parameter value or similar value may vary from a stated amount or range by as much as ±10%.

[0151] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting an intention that the examples claimed require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim existing independently as a separate claimed subject matter.

[0152] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied in many applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A method for controlling an Internet of Things (IoT) product using a portable electronic device including a camera and a display, the method comprising the following steps: capturing frames of video data with the camera; detecting the IoT product at its location in the physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; presenting a virtual control interface on the display according to the IoT product position and the portable electronic device position, wherein the virtual control interface includes a virtual surface virtually presented on the display at a surface position relative to the physical environment; detecting a foot shape performing a foot action at a foot position relative to the physical environment using the captured video data frames; detecting, using the captured video data frame, an intersection between the foot position and the surface position, the intersection and the foot motion corresponding to an IoT motion; and A control signal is sent to the IoT product based on the detected intersection and the foot action, wherein the control signal includes instructions for the IoT product to perform the IoT action.

2. The method according to claim 1, further comprising: The IoT product is paired with the portable electronic device using a product API associated with the IoT product, wherein the product API includes a product type and one or more controllable features.

3. The method according to claim 2, further comprising: The IoT product executes the IoT action, wherein the execution step includes: associating the foot motion with a feature of the one or more controllable features; and The associated features are adjusted according to the foot motion.

4. The method according to claim 2, wherein: The pairing steps also include: A set of possible controllable features is inferred based on the product type.

5. The method according to claim 1, wherein: The determination step also includes: determining whether the IoT product location is within a threshold height of a reference plane in the physical environment, and The presenting step further includes: The virtual control interface is presented adjacent to a virtual plane relative to the reference plane, wherein the virtual control interface includes one or more other virtual surfaces.

6. The method according to claim 1, wherein: The portable electronic device comprises an eyeglass device, the camera comprises an eyeglass camera coupled to the eyeglass device, and the display comprises an eyeglass display coupled to the eyeglass device.

7. The method according to claim 1, wherein: The foot motion comprises one or more movements relative to the foot position, the movement selected from the group consisting of touch and press, hover over, tap and lift, tap and hold, slide and stop, and slide.

8. The method according to claim 1, further comprising: The IoT action is performed by the IoT product by selecting the IoT action based on the detected foot motion.

9. A portable electronic device for controlling an Internet of Things (IoT) product, comprising: processor; Memory; camera; monitor; as well as programming in the memory, wherein execution of the programming by the processor configures the portable electronic device to perform functions including: capturing frames of video data with the camera; detecting the IoT product at its location in the physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; presenting a virtual control interface on the display according to the IoT product position and the portable electronic device position, wherein the virtual control interface includes a virtual surface virtually presented on the display at a surface position relative to the physical environment; detecting a foot shape performing a foot action at a foot position relative to the physical environment using the captured video data frames; detecting, using the captured video data frame, an intersection between the foot position and the surface position, the intersection and the foot motion corresponding to an IoT motion; and A control signal is sent to the IoT product based on the detected intersection and the foot action, wherein the control signal includes instructions for the IoT product to perform the IoT action.

10. The portable electronic device according to claim 9, wherein: Executing the programming further configures the portable electronic device to perform the following functions: The IoT product is paired with the portable electronic device using a product API associated with the IoT product, wherein the product API includes a product type and one or more controllable features.

11. The portable electronic device according to claim 10, wherein: Executing the programming further configures the portable electronic device to perform the following functions: The IoT product performs the IoT action; associating the foot motion with a feature of the one or more controllable features; and The associated features are adjusted according to the foot motion.

12. The portable electronic device according to claim 10, wherein: Executing the programming further configures the portable electronic device to perform the following functions: A set of possible controllable features is inferred based on the product type.

13. The portable electronic device according to claim 9, wherein: Executing the programming further configures the portable electronic device to perform the following functions: determining whether the IoT product location is within a threshold height of a reference plane in the physical environment, and The virtual control interface is presented adjacent to a virtual plane relative to the reference plane, wherein the virtual control interface includes one or more other virtual surfaces.

14. The portable electronic device according to claim 9, wherein: The portable electronic device includes an eyeglass device, the processor includes an eyeglass processor coupled to the eyeglass device, the memory includes an eyeglass memory coupled to the eyeglass device, the camera includes an eyeglass camera coupled to the eyeglass device, and the display includes an eyeglass display coupled to the eyeglass device.

15. The portable electronic device according to claim 9, wherein: Executing the programming further configures the portable electronic device to perform the following functions: Detecting a foot shape performing a foot action, wherein the foot action includes one or more movements relative to the foot position, the movement selected from the group consisting of touching, hovering over, tapping and lifting, tapping and holding, sliding and stopping, and sliding.

16. The portable electronic device according to claim 9, wherein: Executing the programming further configures the portable electronic device to perform the following functions: The IoT action is performed by the IoT product by selecting the IoT action based on the detected foot motion.

17. A non-transitory computer readable medium storing program code, the program code being operable when executed to cause a processor of a portable electronic device to perform the following steps: Capture video data frames with a camera; detecting an Internet of Things (IoT) product at a location of the IoT product in a physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; Presenting a virtual control interface on a display according to the IoT product position and the portable electronic device position, wherein the virtual control interface includes a virtual surface virtually presented on the display at a surface position relative to the physical environment; detecting a foot shape performing a foot action at a foot position relative to the physical environment using the captured video data frames; detecting, using the captured video data frame, an intersection between the foot position and the surface position, the intersection and the foot motion corresponding to an IoT motion; and A control signal is sent to the IoT product based on the detected intersection and the foot action, wherein the control signal includes instructions for the IoT product to perform the IoT action.

18. The non-transitory computer readable medium storing program code according to claim 17, wherein the program code when executed is operable to cause the processor to perform the further steps of: pairing the IoT product with the portable electronic device using a product API associated with the IoT product, wherein the product API includes a product type and one or more controllable features; inferring a set of possible controllable features based on the product type; and The IoT product executes the IoT action, wherein the execution step includes: associating the foot motion with a feature of the one or more controllable features; and The associated features are adjusted according to the foot motion.

19. The non-transitory computer readable medium storing program code according to claim 17, wherein: The portable electronic device comprises an eyeglass device, the processor comprises an eyeglass processor coupled to the eyeglass device, the camera comprises an eyeglass camera coupled to the eyeglass device, and the display comprises an eyeglass display coupled to the eyeglass device.

20. The non-transitory computer readable medium storing program code according to claim 17, wherein the program code when executed is operable to cause the processor to perform the following further steps: Detecting the shape of a foot performing a foot action, wherein The foot motion comprises one or more movements relative to the foot position, the movement selected from the group consisting of touch and press, hover over, tap and lift, tap and hold, slide and stop, and slide.

Images