Controlling virtual AR interface of IOT device using mobile device orientation sensor

Abstract

A virtual AR interface is described for generating a virtual rotation interface to control a connected IoT device using an inertial measurement unit (IMU) of a portable electronic device. The IMU control application enables a user of the portable electronic device to activate a virtual rotation interface overlay on the display and adjust features of the connected IoT product by rotating the portable electronic device. And the device IMU moves the slide block on the virtual rotation interface. The IMU control application sends a control signal to the IoT product, and the IoT product performs an action according to the slider position. The virtual rotation interface is presented on the display as a virtual object in the AR environment. The IMU control application detects device orientation (in a physical environment) and displays in response a corresponding slider element on a virtual rotation interface (in an AR environment).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 901,568, filed on September 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] Examples set forth in this disclosure relate to the field of augmented reality (AR) experiences for electronic devices, including portable electronic devices and wearable devices such as eyewear. More particularly, but not by way of limitation, this disclosure describes an AR application for displaying a virtual rotating interface that enables a user to interact with and control a connected IoT device using an inertial measurement unit (IMU) of a mobile device. Background Art

[0004] Many types of computers and electronic devices available today, such as mobile devices (e.g., smartphones, tablets, and laptops), handheld devices, and wearable devices (e.g., smart glasses, digital glasses, headsets, headbands, and head-mounted displays), include various cameras, sensors, wireless transceivers, input systems, and displays. Users sometimes refer to information on these devices during physical activities such as exercise.

[0005] The so-called "Internet of Things" (IoT) refers to and includes the following physical products that are embedded with sensors, software, and other technologies for connecting and exchanging data with other devices in a network (usually via the Internet). For example, IoT products are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided to act as smart home hubs to connect different smart home products. IoT devices are also used in many other applications. Application layer protocols and supporting frameworks have been provided to implement such IoT applications. For example, some IoT products include an application programming interface (API) that allows IoT products to pair with other products and electronic devices (such as portable computers) and communicate with them separately. Artificial intelligence has also been combined with the IoT infrastructure to achieve more efficient IoT network operation, improve human-computer interaction, and enhance data management and analysis.

[0006] Virtual reality (VR) technology generates a complete virtual environment including realistic images, which is sometimes presented on a VR headset or other head-mounted display. The VR experience allows the user to move around in the virtual environment and interact with virtual objects. AR is a VR technology that combines real objects in the physical environment with virtual objects and displays the combination to the user. The combined display gives the impression that the virtual objects are actually present in the environment, especially when the virtual objects appear and behave like real objects. Cross-reality (XR) is generally understood as an umbrella term for systems that include or combine elements from AR, VR, and MR (mixed reality) environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features of the various examples described will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar element, with additional uppercase or lowercase letters referring to the specific element.

[0008] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:

[0009] Figure 1A is a side view of an eyewear device suitable for use with an example IMU control system (right);

[0010] Figure 1B yes Figure 1A a partial perspective cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0011] Figure 1C yes Figure 1A Side view of the eyewear device (left);

[0012] Figure 1D yes Figure 1C a partial perspective cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0013] Figure 2A and Figure 2B is a rear view of the eyewear device used in the example IMU control system;

[0014] Figure 3 is a block diagram illustrating an example of capturing visible light using the example eyewear device shown in any of the preceding figures;

[0015] Figure 4is a functional block diagram of an example IMU control system including an eyewear device and a server system connected via various networks;

[0016] Figure 5 Is configured for Figure 4 An illustration of an example hardware configuration for a mobile device of an example IMU control system;

[0017] Figure 6 is a stereogram of a user in an example environment, including physical and virtual objects, for reference in describing an example process known as simultaneous localization and mapping (SLAM);

[0018] Figure 7 is a flow chart listing steps in an example method of presenting a virtual rotation interface on a display;

[0019] Fig. 8A is an illustration of an example virtual rotation interface presented on a display of a portable electronic device;

[0020] Figure 8B Portable electronic devices and Fig. 8A , resulting in corresponding IoT actions; and

[0021] Fig. 9 is an illustration of an example interaction between a portable electronic device and an example virtual rotation interface presented on a display of an electronic eyewear device. DETAILED DESCRIPTION

[0022] An IMU control system for use with AR applications on portable electronic devices, including mobile phones and wearable devices such as electronic glasses. The IMU control system enables a user of the portable electronic device to view a virtual rotating interface presented on a display near an IoT product. The user can tilt the electronic device to interact with the virtual rotating interface to adjust one or more controllable features of the IoT product (e.g., turn on, turn off, volume, brightness).

[0023] Various embodiments and details are described with reference to an example for presenting a virtual rotation interface in an augmented reality environment to control an IoT product using an IMU of a portable electronic device having a camera and a display. In one example embodiment, a method involves capturing a video data frame with a camera and detecting an IoT product at an IoT product location in a physical environment using the captured video data frame. The example method also includes determining a portable electronic device position relative to the IoT product position, and then presenting a virtual rotation interface on the display based on the IoT product position and the portable electronic device position. The example virtual rotation interface includes a slider that is virtually presented at a slider position based on the detected device orientation. The slider position corresponds to an IoT action (e.g., adjusting a variable feature such as volume, brightness, or hue). The example method includes sending a control signal to the IoT product, wherein the control signal includes instructions for the IoT product to perform the IoT action.

[0024] Although various systems and methods are described herein with reference to an IMU controlling a slider in a curved viewport, which is typically coplanar with a display, the techniques described herein can be applied to any of a variety of orientation sensors in a portable electronic device that controls a slider or another movable virtual element, whether with or without a viewport or track. In addition, the techniques described herein can be applied to control one or more variable features of a connected IoT product using a portable electronic device moving along or near any rotational plane (e.g., roll, pitch, yaw). For example, a variable feature (e.g., volume) can be controlled by moving the portable electronic device around the z-axis or almost around the z-axis (e.g., yaw). Another feature (e.g., channel selection) can be controlled by moving the portable electronic device around or almost around the y-axis (e.g., roll). Yet another feature (e.g., brightness) can be controlled by moving the portable electronic device around an arbitrary axis (e.g., in any non-orthogonal plane). In this regard, the virtual rotation interface described herein can be used to control various functions associated with IoT consumer products, such as lights, speakers, and fans, as well as IoT products for various commercial, medical, and industrial applications.

[0025] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0026] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0027] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to other parts of the item may be described as "distal." For example, the end of an item closest to the object may be called the proximal end, while the generally opposite end may be called the distal end.

[0028] For purposes of illustration and discussion, the orientation of the eyeglass device, associated components, and any complete device incorporating an eye scanner and camera, such as those shown in any of the accompanying drawings, is given by way of example only. In operation for a particular variable optical processing application, the eyeglass device may be oriented in any other orientation (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eyeglass device. Additionally, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and are not limiting with respect to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.

[0029] Advanced AR technologies, such as computer vision and object tracking, can be used to produce perceptually rich and immersive experiences. Computer vision algorithms extract three-dimensional data about the physical world from data captured in digital images or videos. Object recognition and tracking algorithms are used to detect objects in digital images or videos, estimate their orientation or pose, and track their movement over time. Real-time recognition and tracking of body parts, such as hands and fingers, arms and legs, and feet, is one of the most challenging and processing-intensive tasks in the field of computer vision.

[0030] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized or attained by the methods, instrumentalities and combinations particularly pointed out in the appended claims.

[0031] In a sample configuration, an eyewear device with augmented reality (AR) capabilities is used in the system described herein. Eyewear devices are expected to be used in the system described herein because such devices are scalable and customizable to achieve a personalized experience, achieve the effect of being applied anytime, anywhere, and ensure user privacy by only allowing the user to see the transmitted information. In a sample configuration, the eyewear device can be used without any dedicated hardware, such as the SPECTACLES available from Snap, Inc. of Santa Monica, California. TM .

[0032] like Figures 1A-1D As shown, the eyewear device 100 includes a first camera 114A and a second camera 114B. The cameras 114 capture image information of a scene from separate viewpoints. The captured images can be used to project a three-dimensional display onto an image display for three-dimensional (3D) viewing.

[0033] Cameras 114 are sensitive to wavelengths in the visible light range. Each of cameras 114 defines a different forward-facing field of view that overlaps to enable generation of a three-dimensional depth image, e.g., first camera 114A defines a first field of view 111A, and second camera 114B defines a second field of view 111B. In general, a "field of view" is the portion of a scene visible through a camera at a particular location and orientation in space. Fields of view 111 have overlapping fields of view 304 ( Figure 3 ). When the camera captures an image, objects or object features outside the field of view 111 are not recorded in the original image (e.g., a photograph or picture). The field of view describes the angular range or degree to which the image sensor of the camera 114 picks up electromagnetic radiation of a given scene in the captured image of the given scene. The field of view can be expressed as the angular size of the viewing cone (i.e., the viewing angle). The viewing angle can be measured horizontally, vertically, or diagonally.

[0034] In an example configuration, one or both cameras 114 have a field of view of 100° and a resolution of 480 x 480 pixels. The "coverage angle" describes the range of angles that the lens of the camera 114 can effectively image. Typically, a camera lens produces an image circle large enough to completely cover the film or sensor of the camera, which may include some vignetting (e.g., the image becomes darker toward the edges compared to the center). If the coverage angle of the camera lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective viewing angle will be limited by the coverage angle.

[0035] Examples of suitable cameras 114 include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and digital VGA cameras (video graphics arrays) capable of 480p (e.g., 640 x 480 pixels), 720p, 1080p, or higher resolutions. Other examples include cameras 114 that can capture high-definition (HD) video at high frame rates (e.g., 30 to 60 frames per second or more) and store recordings at a resolution of 1216 by 1216 pixels (or higher).

[0036] The eyewear device 100 can capture image sensor data from the camera 114 and geographic location data digitized by the image processor to be stored in a memory. The camera 114 captures corresponding raw images (e.g., left and right raw images) in a two-dimensional spatial domain, which includes a pixel matrix on a two-dimensional coordinate system, which includes an X-axis for horizontal positioning and a Y-axis for vertical positioning. Each pixel includes a color attribute value (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); and a positioning attribute (e.g., an X-axis coordinate and a Y-axis coordinate).

[0037] To capture stereoscopic images for later display as a 3D projection, image processor 412 ( Figure 4 ) may be coupled to the camera 114 to receive and store visual image information. The image processor 412 or another processor controls the operation of the camera 114 to act as a stereo camera that simulates human binocular vision, and a time stamp may be added to each image. The time stamp on each pair of images allows the images to be displayed together as part of a 3D projection. 3D projection creates an immersive, lifelike experience that is desirable in a variety of environments, including virtual reality (VR) and video games.

[0038] Figure 1B yes Figure 1A 1 is a perspective cross-sectional view of the right corner 110A of the eyewear device 100 depicting the first camera 114A, additional optical components and electronics. Figure 1C yes Figure 1A1 is a side view (left) of an example hardware configuration of the eyewear device 100 showing the second camera 114B of the camera system. Figure 1D yes Figure 1C 1 is a perspective cross-sectional view of a left corner 110B of the eyewear device 100 depicting a second camera 114B, additional optical components, and electronics of the camera system.

[0039] like Figure 1B As shown in the example of FIG. 1 , the eyeglass device 100 includes a first camera 114A and a circuit board 140A, which may be a flexible printed circuit board (PCB). The first hinge 126A connects the right corner 110A to the first temple 125A of the eyeglass device 100. In some examples, the first camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the first temple 125A or the first hinge 126A.

[0040] The right corner 110A includes a corner body 190 and a corner cap. Figure 1B The corner cap is omitted in the cross section of FIG. Disposed within the right corner 110A are various interconnected circuit boards 109, such as a flexible PCB 140A, which include circuit boards for the first camera 114A, (one or more) microphones 139, (one or more) speakers 191, low-power wireless circuit systems (e.g., for communicating via Bluetooth TM wireless short-range network communications), high-speed wireless circuit systems (for example, controller circuits for wireless LAN communications via Wi-Fi).

[0041] The first camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A As shown, the right edge 107A of the eyeglass frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens. The eyeglass frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the eyeglass frame 105. In this example, the first camera 114A has an outward facing field of view 111A (e.g., a line of sight or perspective associated with the right eye of the user of the eyeglass device 100) Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the right corner 110A, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. Coupling may also be achieved indirectly via an intervening component.

[0042] like Figure 1DAs shown in the example of FIG. 1 , the eyeglass device 100 includes a second camera 114B and a circuit board 140B, which may be a flexible printed circuit board (PCB). The second hinge 126B connects the left corner 110B to the second temple 125B of the eyeglass device 100. In some examples, the second camera 114B, the flexible PCB 140B, or other electrical connectors or contacts may be located on the second temple 125B or the second hinge 126B.

[0043] The left corner 110B includes a corner body 190 and a corner cap. Figure 1D The corner cap is omitted in the cross section of FIG. Disposed inside the right corner 110A are various interconnected circuit boards (such as a flexible PCB 140B) that include controller circuitry for the second camera 114B.

[0044] The camera 114 is coupled to or disposed on a corresponding flexible PCB 140 and is covered by a camera cover lens that is aimed through the opening(s) formed in the frame 105. For example, Figure 2A As shown, the right edge 107A of the frame 105 is connected to the right corner 110A and includes (one or more) openings for the camera cover lens, and the left edge 107B of the frame 105 is connected to the left corner 110B and includes (one or more) openings for the camera cover lens. The frame 105 includes a front side that is configured to face outward and away from the user's eyes. The opening for the camera cover lens is formed in and through the front side or outward facing side of the frame 105. In this example, the camera 114 has a corresponding outward facing field of view 111 (e.g., a line of sight or perspective associated with a corresponding eye of a user of the eyewear device 100) Figure 3 ). The camera cover lens may also be adhered to the front side or outward facing surface of the corresponding corner 110, in which the opening is formed with an outward facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intervening component.

[0045] Figure 2A and Figure 2B An example hardware configuration of an eyeglass device 100 including two different types of image displays is depicted. The size and shape of the eyeglass device 100 are configured to be worn by a user. The form of the eyeglasses is shown in the illustrated example. The eyeglass device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0046] In the eyeglass example, the eyeglass device 100 includes a frame 105, which includes a right rim 107A connected to a left rim 107B via a nose bridge 106, and the nose bridge 106 is configured to receive the user's nose to support the eyeglass device 100 on the user's head. The right rim 107A includes a first rim 175A, which holds a first optical element 180A. The left rim 107B includes a second rim 175B, which holds a second optical element 180B. Figure 2B As shown, in some embodiments, each optical element 180A, 180B includes an integrated image display (e.g., a first display 182A and a second display 182B). As used herein, the term "lens" is intended to include a transparent or translucent glass or plastic sheet having a curved or flat surface that causes light to converge or diverge, or to cause little or no convergence or divergence.

[0047] A touch-sensitive input device such as a touch pad 181 is positioned on the first temple 125A. As shown, the touch pad 181 can have a clearly visible border, or include a raised or other tactile edge that provides feedback to the user about the location and border of the touch pad 181; alternatively, the border can be subtle and not easily seen or felt. The eyeglass device 100 can include a touch pad on the other side that operates independently or in conjunction with the touch pad 181.

[0048] The surface of the touch pad 181 is configured to detect finger touches, taps, and gestures (e.g., moving touch) for use with a graphical user interface (GUI) displayed by the eyewear device on the image display to allow the user to browse and select menu options in an intuitive manner, which enhances and simplifies the user experience.

[0049] Detecting finger input on the touchpad 181 can achieve a variety of functions. For example, touching any position on the touchpad 181 can cause the GUI to display or highlight an item on the image display, which can be projected onto at least one of the optical components 180. Tapping or double-clicking the touchpad 181 can select an item or icon. Sliding or swiping a finger in a particular direction (e.g., from front to back, from back to front, from top to bottom, or from bottom to top) can cause an item or icon to slide or scroll in a particular direction; for example, moving to the next item, icon, video, image, page, or slide. Sliding a finger in another direction can slide or scroll in the opposite direction; for example, moving to the previous item, icon, video, image, page, or slide. The touchpad 181 can be positioned substantially anywhere on the eyeglass device 100.

[0050] In one example, a recognized single tap gesture on touch pad 181 initiates selection or pressing of a GUI element in an image presented on the image display of optical assembly 180. Adjusting the image presented on the image display of optical assembly 180 based on the recognized gesture can be a primary action that selects or submits a GUI element on the image display of optical assembly 180 for further display or execution.

[0051] Figure 2A 1 is an example hardware configuration of the eyeglass device 100 in which the right corner 110A supports a microphone 139 and a speaker 191. The microphone 139 includes a transducer that converts sound into a corresponding electrical audio signal. The microphone 139 in the example shown is positioned to have an opening facing inward toward the wearer to facilitate receiving sound waves, such as human speech including verbal commands and questions. Additional or differently oriented openings may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more microphones 139 that are configured to operate together or independently and are positioned at various locations on the eyeglass device 100.

[0052] The speaker 191 includes an electroacoustic transducer that converts an electrical audio signal into a corresponding sound. The speaker 191 is controlled by one of the processors 422, 432 or by the audio processor 413 ( Figure 4 ) control. As shown, the speaker 191 in this example includes a series of rectangular apertures that face inward to direct sound to the wearer. Additional or differently oriented apertures may be implemented. In other example configurations, the eyeglass device 100 is coupled to one or more speakers 191 that are configured to operate together (e.g., stereo, producing surround sound in a zone) or independently and are positioned at various locations on the eyeglass device 100. For example, one or more speakers 191 may be incorporated into the frame 105, temple 125, or corner 110 of the eyeglass device 100.

[0053] Despite Figure 2A and Figure 2B100 is shown as having two optical elements 180, but depending on the application or intended user of the eyeglass device 100, the eyeglass device 100 may include other arrangements, such as a single optical element (or it may not include any optical element 180). As further shown, the eyeglass device 100 includes a right corner 110A adjacent to the right lateral side 170A of the frame 105 and a left corner 110B adjacent to the left lateral side 170B of the frame 105. The corners 110 may be integrated into the frame 105 on the respective sides 170 (as shown), or may be implemented as separate components attached to the frame 105 on the respective sides 170. Alternatively, the corners 110A, 110B may be integrated into temples attached to the frame 105 (not shown).

[0054] In one example, each image display of optical assembly 180 includes an integrated image display (eg, first display 182A and second display 182B). Figure 2A As shown, each optical assembly 180 has a display 182 including a suitable display matrix 177, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, or other such display. Each optical assembly 180 also includes one or more optical layers 176, which may include any combination of lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components. The optical layers (e.g., Figure 2A The optical layer 176A-N (shown as 176A-N in FIG. 104 ) may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eyes. The prisms of the optical layer 176A-N extend over all or at least a portion of the corresponding lens ring 175 formed in the left and right lens edges 107 to allow the user to see the second surface of the prism when the user's eyes look through the corresponding lens edge 107. The first surface of the prism of the optical layer 176A-N faces upward from the frame 105, and the display matrix 177 covers the prism so that photons and light emitted by the display matrix 177 strike the first surface. The prism is designed in size and shape so that light is refracted within the prism and directed toward the user's eyes through the second surface of the prism of the optical layer 176A-N. In this regard, the second surface of the prism of the optical layer 176A-N can be convex to direct light toward the center of the eye. The prisms may optionally be sized and shaped to magnify the image projected by the display matrix 177 , and light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the display matrix 177 .

[0055] In one example, the optical layers 176A-N may include an LCD layer that is transparent (holding the lens open) unless and until a voltage is applied that renders the layer opaque (closing or blocking the lens). The image processor 412 on the eyewear device 100 may be programmed to apply a voltage to the LCD layer, thereby creating an active shutter system, making the eyewear device 100 suitable for viewing visual content when displayed as a 3D projection. Technologies other than LCD may be used for active shutter modes, including other types of reactive layers that respond to a voltage or another type of input.

[0056] In another example, the image display device of the optical assembly 180 has a display 182 that includes a display such as Figure 2B Each optical assembly 180 includes a corresponding laser projector 150, such as a three-color projector using a scanning mirror or a galvanometer. Each laser projector 150 is disposed in or on a corresponding temple 125 of the eyeglass device 100. In this example, each optical assembly 180 includes one or more optical strips (such as Figure 2B 155A-N), these optical strips span the width of the lens of each optical component 180 or span the depth of the lens between the front and back surfaces of the lens and are spaced apart.

[0057] As photons projected by the laser projector 150 travel through the lens of each optical assembly 180, the photons encounter optical strips 155A-N. When a particular photon encounters a particular optical strip, the photon is redirected toward the user's eye, or it passes to the next optical strip. The combination of modulation of the laser projector 150 and modulation of the optical strips controls specific photons or light beams. In an example, the processor controls the optical strips 155A-N by issuing a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical assemblies 180, the eyeglass device 100 may include other arrangements, such as a single or three optical assemblies, or each optical assembly 180 may have a different arrangement depending on the application or intended user of the eyeglass device 100.

[0058] Figure 33D scene 306, a first raw image 302A captured using a first camera 114A, and a second raw image 302B captured using a second camera 114B. As shown, the first field of view 111A can overlap with the second field of view 111B. The overlapping field of view 304 represents the portion of the image captured using the two cameras 114. When referring to the field of view, the term "overlap" means that the pixel matrices in the generated raw images overlap by thirty percent (30%) or more. "Substantially overlap" means that the pixel matrices in the generated raw images or infrared images of the scene overlap by fifty percent (50%) or more. As described herein, the two raw images 302 can be processed to include a timestamp, which allows the images to be displayed together as part of a three-dimensional projection.

[0059] To capture stereo images, such as Figure 3 As shown, a pair of raw red, green, and blue (RGB) images of a 3D scene 306 are captured at a given moment—a first raw image 302A captured using a first camera 114A and a second raw image 302B captured using a second camera 114B. When the pair of raw images 302 are processed (e.g., by an image processor 412), a depth image is generated. The generated depth image can be viewed on an optical assembly 180 of the eyewear device, on another display (e.g., an image display 580 on a mobile device 401), or on a screen.

[0060] The generated depth image is in a three-dimensional spatial domain and may include a matrix of vertices on a three-dimensional position coordinate system including an X-axis for horizontal positioning (e.g., length), a Y-axis for vertical positioning (e.g., height), and a Z-axis for depth (e.g., distance). Each vertex may include a color attribute (e.g., a red pixel light value, a green pixel light value, or a blue pixel light value); a positioning attribute (e.g., an X position coordinate, a Y position coordinate, and a Z position coordinate); a texture attribute; a reflectivity attribute; or a combination thereof. The texture attribute quantifies the perceived texture of the depth image, such as the spatial arrangement of color or intensity in a vertex region of the depth image.

[0061] Figure 4 4 is a functional block diagram of an example IMU control system 400, which includes an eyeglass device 100, a mobile device 401, and a server system 498 connected via various networks 495 (such as the Internet). As shown, the IMU control system 400 includes a low-power wireless connection 425 and a high-speed wireless connection 437 between the eyeglass device 100 and the mobile device 401.

[0062] As described herein, the eyewear device 100 includes one or more cameras 114 that capture still images, video images, or both still images and video images. The camera 114 may have direct memory access (DMA) to the high-speed circuit system 430 and be used as a stereo camera. The camera 114 may be used to capture initial depth images that may be rendered as three-dimensional (3D) models that are texture-mapped images of red, green, and blue (RGB) imaging scenes. The device 100 may also include a depth sensor that uses infrared signals to estimate the position of an object relative to the device 100. In some examples, the depth sensor includes one or more infrared emitters and one or more infrared cameras 410.

[0063] The eyeglass device 100 also includes two image displays (one associated with the right side 170A and one associated with the left side 170B) of the optical assembly 180. The eyeglass device 100 also includes an image display driver 442, an image processor 412, a low power circuit system 420, and a high speed circuit system 430. The image display of the optical assembly 180 is used to present images, including still images, video images, or both still images and video images. The image display driver 442 is coupled to the image display of the optical assembly 180 to control the display of the image.

[0064] Figure 4 The components of the eyeglass device 100 shown in FIG. 1 are located on one or more circuit boards, such as a printed circuit board (PCB) or a flexible printed circuit (FPC) located in the rim or temple. Alternatively or additionally, the depicted components may be located in the corners, frame, hinge, or bridge of the eyeglass device 100. The camera 114 includes a digital camera element, such as a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device, a lens, or any other corresponding visible light or light capturing element that can be used to capture data, including still images or video of a scene with an unknown object.

[0065] like Figure 4 As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In the example, an image display driver 442 is coupled to the high-speed circuit system 430 and is operated by the high-speed processor 432 to drive the image display of the optical assembly 180. The high-speed processor 432 can be substantially any processor capable of managing high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436.

[0066] In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eyeglass device 100, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 also executes a software architecture for the eyeglass device 100, which is used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuit system 436 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to as Wi-Fi herein. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 436.

[0067] The low power circuit system 420 includes a low power processor 422 and a low power wireless circuit system 424. The low power wireless circuit system 424 and the high speed wireless circuit system 436 of the eyeglass device 100 may include a short range transceiver (Bluetooth TM or Bluetooth Low Energy (BLE)) and a wireless wide area network, local area network, or wide area network transceiver (e.g., cellular or Wi-Fi). Mobile device 401 (including a transceiver that communicates via low power wireless connection 425 and high speed wireless connection 437) can be implemented using details of the architecture of eyewear device 100, as can other elements of network 495.

[0068] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, camera data generated by the cameras 114A, 114B, the infrared camera(s) 410, the image processor 412, and images generated for display by the image display driver 442 on the image display of each optical assembly 180. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, separate element of the eyewear device 100. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage addressing of the memory 434 so that the low-power processor 422 initiates the high-speed processor 432 whenever a read or write operation involving the memory 434 is to be performed.

[0069] like Figure 4As shown, various elements of the eyeglass device 100 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both. For example, the infrared camera 410 (including an infrared transmitter in some embodiments), the user input element 491 (e.g., a button switch, a touch pad 181, a microphone 139), and the inertial measurement unit (IMU) 472 can be coupled to the low-power circuit system 420, the high-speed circuit system 430, or both.

[0070] like Figure 5 As shown, CPU 540 of mobile device 401 may be coupled to camera system 570, mobile display driver 582, user input layer 591, and memory 540A, discussed in further detail below.

[0071] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the eyeglass device 100 and the mobile device 401 over the network 495 .

[0072] The output component of the eyewear device 100 includes a visual element, such as a reference Figure 2A and Figure 2B Each lens or optical component 180 described is associated with an image display (e.g., a display such as a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The eyeglass device 100 may include an indicator facing the user (e.g., an LED, a speaker 191, or a vibration actuator) or a signal facing outward (e.g., an LED, a speaker 191). The image display of each optical component 180 is driven by an image display driver 442. In some example configurations, the output components of the eyeglass device 100 also include additional indicators, such as an audible element (e.g., a speaker 191), a tactile component (e.g., an actuator for generating tactile feedback, such as a vibration motor), and other signal generators. For example, the device 100 may include a set of indicators facing the user and a set of signals facing outward. The set of indicators facing the user is configured to be seen or otherwise sensed by the user of the device 100. For example, the device 100 may include an LED display positioned so that the user can see it, one or more speakers 191 positioned to generate sounds that the user can hear, or an actuator that provides tactile feedback that the user can feel. The outwardly facing set of signals is configured to be seen or otherwise sensed by an observer near the device 100. Similarly, the device 100 may include an LED, speaker 191, or actuator configured and positioned to be sensed by an observer.

[0073] The user input elements 491 of the eyewear device 100 may include alphanumeric input components (e.g., a touch screen or touchpad 181 configured to receive alphanumeric input, an optical keyboard, or other alphanumeric configured components), pointer-based input components (e.g., a mouse, touchpad 181, trackball, joystick, motion sensor, or other directional instrument), tactile input components (e.g., a button switch, a touch screen or touchpad 181 or other tactile configured element that senses the location, force, or location and force of a touch or touch gesture), audio input components (e.g., a microphone 139), and the like. The mobile device 401 and the server system 498 may include alphanumeric, pointer-based, tactile, audio, and other input components.

[0074] In some examples, the eyeglass device 100 includes a collection of motion sensing components referred to as an IMU 472. The motion sensing component may be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the IMU 472 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 100 relative to three orthogonal axes (x, y, z) (including acceleration due to gravity). The gyroscope senses the angular velocity of the device 100 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device relative to six axes (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device 100 relative to magnetic north. The position of the device 100 can be determined by a position sensor (such as a GPS unit 473), one or more transceivers for generating relative position coordinates, an altitude sensor or a barometer, and other orientation sensors. Such positioning system coordinates may also be received from mobile device 401 over wireless connections 425 , 437 via low power wireless circuitry 424 or high speed wireless circuitry 436 .

[0075] The IMU 472 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a plurality of useful values ​​about the position, orientation, and motion of the device 100. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the device 100 (in the linear coordinate system x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the device 100 (in the spherical coordinate system). The programming for calculating these useful values ​​may be stored in the memory 434 and executed by the high-speed processor 432 of the eyewear device 100.

[0076] The eyeglass device 100 may optionally include additional peripheral sensors, such as biometric sensors, dedicated sensors, or display elements integrated with the eyeglass device 100. For example, the peripheral device elements may include any I / O components, including output components, motion components, positioning components, or any other such components described herein. For example, the biometric sensor may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identifying people (e.g., recognition based on voice, retina, facial features, fingerprints, or electrical biosignals (such as electroencephalogram data)), etc.

[0077] The mobile device 401 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eyewear device 100 using both a low power wireless connection 425 and a high speed wireless connection 437. The mobile device 401 is connected to a server system 498 and a network 495. The network 495 may include any combination of wired and wireless connections.

[0078] like Figure 4 As shown, the IMU control system 400 includes a computing device, such as a portable electronic device 401 (e.g., a mobile device), coupled to the eyewear device 100 via a network 495. The IMU control system 400 includes a memory (e.g., a non-transitory computer-readable medium) for storing instructions and a processor for executing instructions. In some embodiments, the memory and processing functions of the IMU control system 400 can be shared or distributed between the processors and memories of the eyewear device 100, the portable electronic device 401 (e.g., a mobile device), and / or the server system 498.

[0079] In some implementations, the IMU control system 400 includes one or more elements or modules, referred to herein as an IMU control application 910 , a positioning system 915 , an image processing system 920 , and a speech recognition module 925 .

[0080] As described herein, in some implementations, the IMU control application 910 renders and presents the virtual rotation interface 800 on a display.

[0081] In some embodiments, the positioning system 915 obtains positioning data for determining the current position of the portable electronic device 401 (e.g., a mobile phone) or the electronic eyewear device 100 relative to the physical environment 600. For example, the positioning system 915 can access the video data frame 900 captured using the camera 114B to determine the current eyewear device position 840 relative to the physical environment 600 in three-dimensional coordinates (with or without reference to data from other sources, such as an inertial measurement unit or IMU 472). As used herein, the term "video data frame" refers to video motion data captured using one or more cameras 114A, 114B coupled to the eyewear device 100, including images, spatial data, and related information captured using substantially any sensor component of the camera in any form and at any sampling rate. In some embodiments, the positioning data can be derived from the motion data frame captured using the IMU 472, from data collected by the GPS unit 473, or from a combination thereof.

[0082] In some embodiments, the image processing system 920 cooperates with the image display driver 442 and the image processor 412 to generate and present a virtual rotation interface 800 as described herein on a display of a portable electronic device 401 (e.g., a mobile phone) or an electronic eyewear device 100. In some embodiments, the virtual rotation interface 800 includes one or more virtual surfaces, each of which is virtually presented at a surface location 810 in the physical environment 600 relative to the display.

[0083] In some implementations, the speech recognition module 925 receives human speech, converts the received speech into audio data frames, recognizes a query or request based on the audio data, and performs actions associated with and responsive to the recognized query or request.

[0084] Figure 5 is a high-level functional block diagram of an example mobile device 401. Mobile device 401 includes flash memory 540A that stores programming to be executed by CPU 540 to perform all or a subset of the functionality described herein.

[0085] Mobile device 401 may include camera 570, which includes at least two cameras (e.g., a first visible light camera and a second visible light camera with overlapping fields of view) or at least one camera and a depth sensor with substantially overlapping fields of view. Flash memory 540A may also include multiple images or videos generated via camera 570.

[0086] As shown in the figure, the mobile device 401 includes an image display 580, a mobile display driver 582 for controlling the image display 580, and a display controller 584. Figure 5In the example of , image display 580 includes a user input layer 591 (eg, a touch screen) that is layered over or otherwise integrated into the screen used by image display 580 .

[0087] Examples of touch screen mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, therefore, Figure 5 A block diagram illustration of an example mobile device 401 is provided with a user interface including a touch screen input layer 591 for receiving input (through touch, multi-touch, gestures, etc., by hand, stylus, or other implement) and a graphics display 580 for displaying content.

[0088] like Figure 5 As shown, the mobile device 401 includes at least one digital transceiver (XCVR) 510, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 401 also includes additional digital or analog transceivers, such as for short-range network communication (such as via NFC, VLC, DECT, ZigBee, Bluetooth TM For example, the short-range XCVR 520 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as one of the Wi-Fi standards under IEEE 802.11.

[0089] To generate location coordinates for positioning the mobile device 401, the mobile device 401 may include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 401 may utilize either or both of the short-range XCVR 520 and the WWAN XCVR 510 to generate location coordinates for positioning. For example, based on a cellular network, Wi-Fi, or Bluetooth TM The positioning system of the present invention can generate accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the eyewear device via XCVR 510, 520 through one or more network connections.

[0090] In some examples, the mobile device 401 includes a collection of motion sensing components called an inertial measurement unit (IMU) 572 for sensing the position, orientation, and motion of the mobile device 401. The motion sensing component can be a microelectromechanical system (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the inertial measurement unit (IMU) 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the mobile device 401 relative to three orthogonal axes (x, y, z) (including the acceleration caused by gravity). The gyroscope senses the angular velocity of the mobile device 401 around three rotational axes (pitch, roll, yaw). The accelerometer and gyroscope can provide together the position, orientation, and motion data of the device relative to six degrees of freedom (x, y, z, pitch, roll, yaw) (sometimes referred to as 6DOF). The magnetometer (if present) senses the heading of the mobile device 401 relative to magnetic north.

[0091] The IMU 572 may include or cooperate with a digital motion processor or programming that collects raw data from the components and calculates a number of useful values ​​about the position, orientation, and motion of the device mobile device 401. For example, the acceleration data collected from the accelerometer may be integrated to obtain the velocity relative to each axis (x, y, z); and integrated again to obtain the position of the mobile device 401 (in the linear coordinate system x, y, and z). The angular velocity data from the gyroscope may be integrated to obtain the position of the mobile device 401 (in the spherical coordinate system). The programming for calculating these useful values ​​may be stored in one or more memory elements 540A, 540B, 540C and executed by the CPU 540 of the mobile device 401.

[0092] The transceivers 510, 520 (i.e., network communication interfaces) conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 510 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). For example, transceivers 510, 520 provide two-way wireless communication of information (including digitized audio signals, still images and video signals, web page information for display, and web-related input), as well as various types of mobile message communications to / from mobile device 401.

[0093] The mobile device 401 also includes a microprocessor that acts as a central processing unit (CPU), such as Figure 4 540 in . A processor is a circuit having the following elements that are structured and arranged to perform one or more processing functions, typically performing various data processing functions. Although discrete logic components can be used, the example utilizes components that constitute a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that combine electronic components for performing the functions of a CPU. For example, CPU 540 can be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuit systems can be used to constitute CPU 540 or processor hardware in smart phones, laptop computers, and tablet computers.

[0094] The CPU 540 acts as a programmable main controller for the mobile device 401 by configuring the mobile device 401 to perform various operations, for example, according to instructions or programming executable by the CPU 540. For example, such operations may include various general operations of the mobile device, as well as operations related to programming for applications on the mobile device. Although the processor can be configured by using hard-wired logic, a typical processor in the mobile device is a general-purpose processing circuit that is configured by executing programming.

[0095] The mobile device 401 includes a memory or storage system for storing programming and data. In this example, the memory system may include flash memory 540A, random-access memory (RAM) 540B and other memory components 540C as needed. RAM 540B serves as short-term storage for instructions and data processed by CPU 540, for example, as working data processing memory. Flash memory 540A typically provides long-term storage.

[0096] Thus, in the example of mobile device 401, flash memory 540A is used to store programming or instructions for execution by CPU 540. Depending on the type of device, mobile device 401 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or similar systems.

[0097] The processor 432 in the eyeglass device 100 can construct a map of the environment surrounding the eyeglass device 100, determine the position of the eyeglass device in the map of the environment, and determine the relative positioning of the eyeglass device and one or more objects in the mapped environment. The processor 432 can use a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors to construct a map and determine the position and positioning information. The sensor data includes images received from one or two of the cameras 114A, 114B, (one or more) distances received from the laser rangefinder, positioning information received from the GPS unit 473, motion and acceleration data received from the IMU 572, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information. In the context of augmented reality, the SLAM algorithm is used to construct and update a map of the environment while tracking and updating the position of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods (such as particle filters, Kalman filters, extended Kalman filters, and covariance intersection). In a system including a high-definition (HD) camera that captures video at a high frame rate (e.g., 30 frames per second), the SLAM algorithm updates the map and the position of objects at least as frequently as the frame rate; in other words, the map and localization are calculated and updated 30 times per second.

[0098] The sensor data includes one or more images received from one or both cameras 114A, 114B, (one or more) distances received from a laser rangefinder, positioning information received from a GPS unit 473, motion and acceleration data received from an IMU 472, or a combination of data from such sensors or from other sensors that provide data that helps determine positioning information.

[0099] Figure 6 An example physical environment 600 and elements useful when using SLAM algorithms (simultaneous localization and mapping) and other types of tracking applications (e.g., natural feature tracking (NFT), hand tracking, foot tracking, etc.) are depicted. A portable electronic device 401 (e.g., a mobile phone) and a user 602 of the electronic eyewear device 100 are present in the example physical environment 600 (in Figure 6, the example physical environment is an indoor room). The processor 432 of the eyeglass device 100 (or the processor of the mobile phone 401) uses the captured image data to determine the position of the electronic device 100, 401 relative to one or more physical objects 604 (e.g., including the IoT object 50) within the environment 600, constructs a map of the environment 600 using a coordinate system of the environment 600 (e.g., a Cartesian coordinate system (x, y, z)), and determines the position of the device 100, 410 relative to the coordinate system. In addition, in some embodiments, the processor determines the precise orientation and position (x-prime, y-prime, z-prime, roll, pitch, and yaw) of the portable electronic device 401 or the eyeglass device 100 in the environment by using two or more position points (e.g., three position points 606a, 606b, and 606c) associated with a single object 604a, or by using one or more position points 606 associated with two or more objects 604a, 604b, 604c. An application on the portable electronic device 401 or the eyewear device 100 can present a virtual object 608 (such as Figure 6 keys shown) to view them on the display during the augmented reality experience.

[0100] In some examples, the positioning system 915 includes a virtual marker 610a associated with a virtual object 608 in the physical environment 600. In an augmented reality environment, in some embodiments, markers are registered at locations in the physical environment 600 to assist the electronic device in the task of tracking and updating the location of users, devices, and objects (virtual and physical) relative to the physical environment. Markers are sometimes registered to high-contrast physical objects (such as relatively dark objects mounted on a light-colored wall, such as framed pictures 604a) to assist cameras and other sensors in the task of detecting markers. Markers can be assigned and registered in memory by the eyewear device 100 operating in the environment. In some embodiments, markers are assigned and registered in the memory of other devices in the network.

[0101] The positioning system 915 tracks physical objects and virtual objects within the physical environment 600 relative to the eyeglass device 100. For physical objects 604 (e.g., example floor safe 604c, example IoT object 50), the positioning system 915 continuously analyzes the captured images of the physical environment 600 to identify the object 604 and determine its position relative to the eyeglass device 100 (e.g., by applying a SLAM algorithm). The positioning system 915 maintains and updates the determined position information of the physical object 604 in a memory so that the physical object 604 is tracked as the eyeglass device 100 moves through the physical environment 600. For virtual objects 608 (e.g., keys), the positioning system 915 establishes or specifies an initial position of the virtual object 608 corresponding to a position in the environment 600 or the physical object 604 (or, in some embodiments, a position relative to the eyeglass device 100). The positioning system 915 maintains and updates position information of the virtual object 608, such as according to a movement algorithm associated with the virtual object 608, in response to the eyeglass device 100 moving through the environment, or a combination thereof, thereby tracking the virtual object 608 as the eyeglass device 100 moves through the environment.

[0102] The marker may be encoded with information or otherwise linked to information. The marker may include positioning information, a physical code (such as a barcode or QR code; visible or hidden to the user), or a combination thereof. A set of data associated with the marker is stored in the memory 434 of the eyewear device 100. The set of data includes information about the marker 610a, the positioning (position and orientation) of the marker, one or more virtual objects, or a combination thereof. The marker positioning may include three-dimensional coordinates of one or more marker landmarks 616a, such as Figure 6 The corners of the roughly rectangular marker 610a shown in . The marker location can be expressed relative to real-world geographic coordinates, a marker coordinate system, the positioning of the eyewear device 100, or other coordinate systems. One or more virtual objects associated with the marker 610a can include any of a variety of materials (including still images, videos, audio, tactile feedback, executable applications, interactive user interfaces, and experiences) and combinations or sequences of such materials. In this context, any type of content that can be stored in memory and retrieved when the marker 610a is encountered or associated with an assigned marker can be classified as a virtual object. For example, Figure 6 The virtual key 608 shown in FIG. 6 is a virtual object displayed as a 2D or 3D still image at a marked location.

[0103] In one example, the marker 610a can be registered in memory as being located on the physical object 604a (e.g., Figure 6 In another example, the marker may be registered in memory as a specific location relative to the eyewear device 100.

[0104] Figure 7 700 is a flowchart of an example method of controlling an IoT product 50 using a portable electronic device 401. Although these steps are described with reference to the portable electronic device 401 (e.g., a mobile phone) or the eyewear device 100 described herein, those skilled in the art will understand other implementations of the steps for other types of devices based on the description herein. One or more of the steps shown and described may be performed simultaneously, in series, in an order different from that shown and described, or in combination with additional steps. Certain steps may be omitted or repeated in certain applications.

[0105] In some embodiments, the IMU control application 910 described herein is launched in response to receiving a selection through a user interface (e.g., selection from a menu, pressing a button, using a touchpad) or through some other input method (e.g., a gesture detected in a captured image, a finger touch 681 on the touchpad 181, a voice command).

[0106] In some embodiments, an IoT product 50 (e.g., a light, a speaker, a fan) is paired with a portable electronic device 401 or an electronic eyeglass device 100. Some IoT products 50 include an application programming interface (API) (e.g., a product API 52) that allows the IoT product 50 to be paired with and otherwise communicate with other products and electronic devices (such as the portable electronic device 401 and the electronic eyeglass device 100). In some embodiments, the product API 52 includes information such as the product type (e.g., a light, a speaker, a fan), and in some embodiments, a list of one or more controllable features 56 associated with the IoT product 50 (e.g., on, off, brightness, color temperature, tint, volume, speed).

[0107] Fig. 8A is an illustration of an example virtual rotation interface 800 presented on the display 580 of the portable electronic device 401. In this example, the IoT product 50 is a lamp, as shown. Of course, the IoT product 50 exists in the physical environment 600 and is observable. When the IMU control application 910 described herein is presented on the display 580, the virtual rotation interface 800 is observable.

[0108] In some embodiments, the IMU control application 910 controls the pairing process of the IoT product 50 with the portable electronic device 410. In some embodiments, the pairing process uses a product API. In some embodiments, the IoT product 50 includes an associated product API that includes a product type (e.g., "light") and a list of one or more controllable features 56 (e.g., brightness, color temperature, tint).

[0109] Figure 8B is a portable electronic device 401 and Fig. 8A , resulting in a corresponding IoT action 60 (e.g., lowering the brightness of a light).

[0110] Fig. 9 is an illustration of an example interaction between a portable electronic device 401 and an example virtual rotation interface 800 presented on a display 182B of an electronic eyewear device 100. In some embodiments, the eyewear device 100 includes a speech recognition module 925 and a microphone 139 coupled to a speaker 191 as described herein. In some embodiments, the speech recognition module 925 configures the processor 432 to perceive human speech, convert the received speech into an audio data frame, identify a first query based on the converted audio data frame, and then perform an action in response to and according to the identified first query. For example, the human speech may include a verbal command (e.g., "show the virtual rotation interface"), and in response, the identified first query causes the IMU control application 910 to present the virtual rotation interface 800 on the display.

[0111] In some embodiments, the IoT product 50 does not include an API. The IoT product 50 is typically connected to other products in the Internet or network. Due to this connection, the communication between and among the IoT products 50 typically includes data or information indicating the product type (e.g., lamp, speaker, fan). In this regard, even when the IoT product 50 does not include a product API, the product type is generally known. In some embodiments, the IMU control application 910 described herein includes an inference engine that is configured to infer a set of possible controllable features 56 associated with the IoT product 50 based solely on the product type. For example, the IoT product 50 may be sharing data indicating that the product type is "speaker". In some embodiments, the inference engine will infer a set of possible controllable features 56 (e.g., on, off, variable volume, available natural sound list) because these features are typically associated with specific product types (e.g., speakers or sound systems). For another example, when the product type is "fan", in some embodiments, the inference engine will infer a set of possible controllable features 56 (e.g., an on-off toggle switch for power, variable fan speed (from minimum to maximum), a set of predefined fan speeds (low, medium, high), and an on-off toggle switch for oscillation (stationary or moving), etc.). When the product type is "LED light", in some embodiments, the inference engine will infer a set of possible controllable features 56 (e.g., an on-off toggle switch for power, variable brightness (from minimum to maximum), variable color selection (in a continuous cycle of hues or colors), etc.).

[0112] Figure 7 Block 702 in the example method steps of capturing a video data frame 900 with the camera 570 of the portable electronic device 401 is described. In some embodiments, the process of capturing the video data frame 900 is ongoing during active use of the portable device 401. In other examples, the capture process begins in response to receiving a selection through a user interface or through some other input means. In some embodiments, the example method steps at block 702 include storing the captured video data frame 900 at least temporarily in a memory on the portable electronic device 401 so that the video data frame 900 can be used for purposes including processing and analysis.

[0113] Figure 7 Block 704 in the example steps of detecting the IoT product 50 at the IoT product location 790 in the physical environment 600 based on the captured video data frame 900. As described herein, in some embodiments, the IoT product 50 has been paired with or otherwise wirelessly communicated with the portable electronic device 401. In this regard, the detection process occurs when (and only if) the IoT product 50 is within a threshold proximity of the portable electronic device 401.

[0114] Fig. 8A An example IoT product 50 is shown located in a physical environment 600 at an IoT product location 790 .

[0115] In some embodiments, the process of detecting the IoT product 50 includes determining or estimating the IoT product location 790 relative to the physical environment 600. In some embodiments, the current IoT product location 790 is estimated using the positioning system 915 described herein.

[0116] In some embodiments, the positioning system 915 configures the IMU control application 910 to obtain positioning data based on the captured video data frames 900, and in some embodiments, based on motion data collected by an inertial measurement unit (IMU). In some embodiments, the positioning system 915 uses a SLAM algorithm to build a virtual map of one or more objects within the camera field of view 904, as described herein, and the SLAM algorithm updates the map and the positions of the objects at least as frequently as the camera captures the video data 900.

[0117] In some embodiments, the process of determining or estimating the IoT product location 790 relative to the physical environment 600 includes calculating a correlation between the IoT product location 790 and the current portable electronic device location 440 (e.g., Fig. 8A). The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the IoT product location 790 and the current portable electronic device location 440. Because the positioning process occurs frequently, the correlation process between the IoT product location 790 and the portable electronic device location 440 produces accurate and near real-time tracking of the IoT product location 790 relative to the portable electronic device location 440.

[0118] Block 706 describes example steps of using the captured video data frame 900 to determine the portable electronic device location 440 relative to the IoT product location 790. As the portable electronic device 401 moves through the physical environment 600, its location relative to the IoT product location 790 changes.

[0119] Fig. 8A Example portable electronic device 410 is shown in physical environment 600 located at portable electronic device location 440 .

[0120] In some implementations, portable electronic device location 440 is estimated using positioning system 915 as described herein.

[0121] In some embodiments, the process of determining or estimating the portable electronic device location 440 relative to the IoT product location 790 includes calculating a correlation between the two locations. The term correlation refers to and includes one or more vectors, matrices, formulas, or other mathematical expressions sufficient to define the three-dimensional distance between the current portable electronic device location 440 and the current IoT product location 790. Because the positioning process occurs frequently, the correlation process produces accurate and near real-time tracking of the portable electronic device location 440 relative to the IoT product location 790. In some embodiments, the process of estimating the current portable electronic device location 440 is based on motion data frames captured using an IMU, or based on video data frames 900 captured using a camera, or a combination of both.

[0122] Figure 7 Block 708 in describes example steps for using the IMU 572 to detect the device orientation 80 relative to the physical environment 600 . Fig. 8A An example portable electronic device 401 is shown having a device orientation 80a relative to a physical environment 600 . Figure 8B Device 401 is shown in a new device orientation 80b relative to physical environment 600 . Fig. 9 An example eyewear device 100 is shown having a device orientation 80c relative to a physical environment 600 .

[0123] In some embodiments, the IMU 572 is used alone in the process of detecting the device orientation 80. In contrast, the captured video data frame 900 is used in the process of block 706 of determining the portable electronic device position 440. In some cases, the device orientation 80 and the portable electronic device position 440 are nearly identical. Both values ​​80, 440 include three-dimensional coordinates.

[0124] In some cases (e.g., when device orientation 80 or IMU data is not available), the process of detecting device orientation 80 includes selecting and using portable electronic device location 440 as device orientation 80. Similarly, in some cases (e.g., when portable electronic device location 440 or video data 900 is not available), device orientation 80 may be used as portable electronic device location 440.

[0125] As described herein, in some embodiments, the IMU 572 on the portable electronic device 401 (and the IMU 372 of the eyeglass device 100) includes a collection of motion sensing components. The motion sensing components can be microelectromechanical systems (MEMS) with microscopic moving parts (usually small enough to be part of a microchip). In some example configurations, the IMU 472, 572 includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer senses the linear acceleration of the device 401, 100 relative to three orthogonal axes (x, y, z) (including the acceleration caused by gravity). The gyroscope senses the angular velocity of the device 401, 100 around three rotation axes (pitch, roll, yaw). The accelerometer and gyroscope can together provide position, orientation, and motion data about the device 401, 100 relative to six degrees of freedom (x, y, z, pitch, roll, yaw). The magnetometer (if present) senses the heading of the device relative to magnetic north.

[0126] In some embodiments, the process of detecting device orientation 80 includes detecting a starting device orientation 81 (e.g., Fig. 8A , generally vertically), and associates the starting device orientation 80 with a current value of one of the controllable features 56 (e.g., brightness). Fig. 8A In the example shown, the example current value is exactly 100%; however, this is not always the case. In some embodiments, the process of detecting device orientation 80 includes presenting a highlight or other indicia near slider 802 when the starting device orientation 81 is associated with the current value of the feature. In this example, the user is alerted that portable device 401 is engaged with slider 802. In some embodiments, the process of detecting starting device orientation 81 occurs when electronic device position 440 is first detected within a threshold proximity of IoT product position 790.

[0127] In this example, the process of detecting the device orientation 80 also includes detecting a final device orientation 89 (eg, a clockwise tilt, such as Figure 8B ), and associates the final device orientation 89 with the IoT action 60. In some embodiments, the process of detecting the final device orientation 89 occurs when the electronic device position 440 is no longer detected within a threshold proximity of the IoT product position 790. In other embodiments, the process of detecting the final device orientation 89 occurs in response to user input.

[0128] Figure 7 Block 810 in describes example steps for presenting a virtual rotation interface 800 on the display 580 based on the IoT product location 790 and the portable electronic device location 440. In this example, the virtual rotation interface 800 is presented at a consistent location on the display 580 relative to the IoT product location 790 even when the portable electronic device 401 moves through the physical environment 600. For example, Fig. 8A As shown, the virtual rotation interface 800 is presented at a viewport location 812 on the display 580 (eg, near the top or upper portion of the IoT lamp 50 ).

[0129] In some embodiments, the process of presenting the virtual rotation interface 800 includes moving or otherwise adjusting the viewport position 812 relative to the IoT product 50. In some embodiments, the IMU control application 910 moves or otherwise adjusts the viewport position 812 so that the virtual rotation interface 800 will be presented in the foreground relative to the IoT product 50. In some embodiments, the IMU control application 910 automatically moves or otherwise adjusts the viewport position 812 without user input. In some embodiments, the viewport position 812 is configurable so that the process of presenting the virtual rotation interface 800 includes providing the user with one or more tools to adjust or otherwise configure the viewport position 812.

[0130] In some embodiments, the process of presenting the virtual rotation interface 800 includes providing the user with one or more tools to adjust or otherwise configure the virtual rotation interface 900, as described herein, or to selectively start and stop presentation. For example, the user can pause or stop (or restart) the process of presenting the virtual rotation interface 800 by speaking a voice command, pressing a button on the eyewear device 100, performing a gesture, or tapping a finger touch 681 on the touchpad 181.

[0131] In some implementations, the virtual rotation interface 800 includes a slider 802 that is virtually rendered at a slider position 804 based on a detected device orientation 80, such as Figure 8B For example, in Figure 8B, the slider position 804 (eg, at a position along the viewport 806, with Fig. 8A The starting position (clockwise distance shown) corresponds to a detected device orientation 80 (eg, the mobile device 401 has been rotated clockwise relative to its starting orientation).

[0132] In some embodiments, Fig. 8A As shown, the virtual rotation interface 800 includes a plurality of sliders 802 (eg, slider 802a located on the right viewport 806 and slider 802b located on the left viewport). The arched viewport in this example is presented as a semicircle around the top of the IoT lamp 50.

[0133] In some implementations, presenting virtual rotation interface 800 includes presenting slider 802 (and viewport 806, if present) as an overlay relative to physical environment 600, such that virtual rotation interface 800 is continuously visible in the foreground relative to real objects in physical environment 600. For example, Fig. 8A As shown, the virtual rotation interface 800 is presented as an overlay (e.g., in the foreground, in front of the lamp 50) relative to the physical environment 600 (e.g., the lamp 50 and the wall behind it), so that the virtual rotation interface 800 is continuously visible relative to other objects in the physical environment 600.

[0134] In some embodiments, the process of presenting the virtual rotation interface 800 includes rendering or otherwise graphically generating the slider 802 (and the viewport 806, if present) in a predefined and configurable size and shape. Fig. 8A As shown, the virtual rotation interface 800 is presented at a size that nearly fills the available space on the display 580. In some embodiments, the relative size of the virtual rotation interface 800 automatically changes based on the IoT product position 790, the portable electronic device position 440, and the available space on the display 580.

[0135] In some embodiments, presenting the virtual rotation interface 800 includes presenting a marker on the display 580 near the slider 802. In some embodiments, the marker is associated with one or more controllable features 56. For example, Fig. 8A , the mark includes a window showing the 100% value. Figure 8B As shown, the marker includes a window showing the 59% value.

[0136] In some embodiments, the process of presenting the virtual rotation interface 800 includes presenting the interface 800 when the electronic device position 440 is within a threshold proximity (e.g., mathematically evaluating the distance between the electronic device position 440 and the IoT product position 790) of the IoT product 50. In this example, the virtual rotation interface 800 appears when the electronic device 401 is close to the IoT product 50.

[0137] In some embodiments, the process of presenting the virtual rotation interface 800 includes estimating the device rotation plane 820 using the IMU 572. The device rotation plane 820 generally corresponds to the rotation plane detected by the IMU 572 inside the portable device 401. In some embodiments, the device rotation plane 820 is the average or median of a series of values ​​collected by the IMU 572 over time. Figure 8B , the device rotation plane 820 is graphically represented by a circular arrow near the top edge of the device 401. In some embodiments, the process at block 710 includes presenting the virtual rotation interface 800 according to the estimated device rotation plane 820 such that the interface 800 appears aligned with the device rotation plane 820 on the display 580 (e.g., as the portable device 410 rotates over time).

[0138] Figure 7 Block 712 in describes an example step of sending a control signal 55 to an IoT product 50 based on the device orientation 80. In some embodiments, the process of sending the control signal 55 includes sending the control signal 55 to the IoT product 50 using a wireless transceiver, sending the control signal 55 to a cloud-based service, which in turn transmits the control signal 55 to the IoT product 50, or otherwise broadcasting the control signal 55 using methods and means to facilitate delivery of the control signal 55 to the IoT product 50.

[0139] In some embodiments, the control signal 55 includes instructions configured for use by the IoT product 50 to facilitate implementation or execution of an associated IoT action 60. For example, when the device orientation 80 (e.g., rotated clockwise relative to vertical) corresponds to a particular IoT action 60 (e.g., reducing brightness), the control signal 55 includes instructions for the IoT product 50 to perform the IoT action 60 (e.g., reduce brightness accordingly).

[0140] Figure 7 Block 714 in describes example steps for the IoT product 50 to perform or otherwise execute the IoT action 60 according to the device orientation 80. In some implementations, the control signal 55 includes instructions for the IoT product 50 to perform or otherwise execute the IoT action 60.

[0141] Figure 7Block 716 in describes example steps for establishing and calibrating the virtual ruler 810. In some embodiments, the IoT product 50 includes or is associated with a product API as described herein. In some embodiments, the product API includes a controllable feature 56 (e.g., brightness) and a range 58 of variable values ​​59 (e.g., from zero to one hundred percent and from one to eleven, etc.).

[0142] In some embodiments, the virtual ruler 810 extends from a low end 811 to a high end 819. For example, Figure 8B The viewport 806 is shown extending in a clockwise direction from a low end 811 toward a high end 819. In this example, the low end 811 represents a maximum brightness of 100%, and the high end 819 represents a minimum brightness.

[0143] In some embodiments, the process of calibrating the virtual ruler 810 (e.g., low end 811 to high end 819) includes comparing the virtual ruler 810 to a range of values ​​59 (e.g., one hundred to zero), and mathematically correlating the length of the virtual ruler 810 (e.g., ninety degrees) to the range of values ​​59 (e.g., one hundred to zero). In this example, mathematically, each 9 degree increment along the virtual ruler 810 is associated with a 10% increment in brightness value 59.

[0144] These steps are described with reference to a portable electronic device 401 (e.g., a mobile phone). In some embodiments, the methods and systems described herein can be executed and operated on the eyeglass device 100, the mobile phone 401, or another electronic device. For example, in some embodiments, the portable electronic device 401 is an electronic eyeglass device 100 as described herein. In this example, the camera 570 is the eyeglass camera 114B, the display 580 is the eyeglass display 182B, the IMU 572 is the eyeglass IMU 472, the memory elements 540A, 540B, 540C are the eyeglass memory 434, etc.

[0145] In some embodiments, the methods and systems described herein are processed and executed using a portable electronic device 401 (e.g., a mobile phone), and the virtual rotation interface 800 is presented on the display 580 of the mobile phone 401, or on the display 182B of the eyeglass device 100, or both. In this example, the IMU of the mobile phone 401 controls the device orientation 80 and the slider 802, and the virtual rotation interface 800 can be observed by a person wearing the electronic eyeglass device 100.

[0146] Any functions of the eyewear device 100, mobile device 401, and server system 498 described herein may be embodied in one or more computer software applications or programming instruction sets, as described herein. According to some examples, a "function," "multiple functions," "application," "multiple applications," "instruction," "multiple instructions," or "programming" is a program (or programs) that executes the functions defined in the program. Various programming languages ​​may be used to generate one or more applications that are structured in a variety of ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , Mobile software running on a mobile phone or other mobile operating system. In this example, the third-party application can call API calls provided by the operating system to facilitate the functions described in this article.

[0147] Thus, machine-readable media can take the form of tangible / non-temporary storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer device or the like, such as can be used to implement the client device, media gateway, transcoder, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that contain buses within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common computer-readable media forms include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other medium from which a computer can read programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions / program codes to a processor for execution.

[0148] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.

[0149] It will be understood that the terms and expressions used herein have the common meanings given by such terms and expressions relative to their corresponding corresponding explorations and research fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second, etc.) can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only includes those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0150] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range that is consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, a parameter value or similar value may vary from a stated amount or range by as much as ±10%.

[0151] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are combined together in various examples. This method of disclosure should not be interpreted as reflecting such an intention that the examples claimed for protection require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim existing independently as a separate claimed subject matter.

[0152] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied in many applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A method for controlling an Internet of Things (IoT) product using a portable electronic device, the portable electronic device comprising an inertial measurement unit (IMU), a camera, and a display, the method comprising: capturing frames of video data with the camera; detecting the IoT product at its location in the physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; detecting an orientation of the device relative to the physical environment using an IMU; presenting a virtual rotation interface on the display at a viewport position based on the IoT product position and the portable electronic device position, wherein the virtual rotation interface includes: a slider virtually presented at a slider position based on the detected device orientation, and wherein the slider position corresponds to an IoT action; and A control signal is sent to the IoT product based on the detected device orientation, wherein the control signal includes instructions for the IoT product to perform the IoT action.

2. The method according to claim 1, further comprising: The IoT product executes the IoT action according to the instruction.

3. The method according to claim 1, wherein: The IoT product includes a controllable feature represented by a current value, wherein presenting the virtual rotation interface further includes: When the portable electronic device is located within a threshold proximity of the IoT product, presenting the virtual rotation interface; and Among them, the detection equipment orientation also includes: detecting a starting device orientation when the portable electronic device position is first detected within the threshold proximity; associating the starting device orientation with a current value of the controllable characteristic; detecting a final device orientation when the portable electronic device position is no longer detected within the threshold proximity; and The final device orientation is associated with the IoT action.

4. The method according to claim 3, further comprising: A final device orientation is detected in response to the user input.

5. The method according to claim 1, wherein: Presenting the virtual rotation interface further includes: presenting the slider relative to a viewport; presenting additional sliders virtually rendered at additional slider positions relative to the additional viewport, wherein the additional slider positions correspond to additional IoT actions; In response to user input, associating the detected device orientation with at least one of the slider and the additional slider; and At least one of the slider and the additional slider is presented based on the association and the detected device orientation.

6. The method according to claim 1, wherein: The IoT product is associated with a product API, the product API including a product type and controllable features associated with a variable value range, and wherein presenting the virtual rotation interface further comprises: establishing a virtual ruler relative to the orientation of the device, the virtual ruler extending angularly clockwise from a low end to a high end; calibrating the virtual ruler using the variable value range; and The slider is rendered relative to a viewport that is virtually rendered according to the calibrated virtual ruler.

7. The method according to claim 1, wherein: Presenting the virtual rotation interface further includes: estimating a device rotation plane using the IMU; and The virtual rotation interface is presented according to the estimated device rotation plane.

8. The method according to claim 1, further comprising: The IoT action is performed by the IoT product by selecting the IoT action based on the detected device orientation.

9. A portable electronic device for controlling an Internet of Things (IoT) product, comprising: processor; Memory; Inertial Measurement Unit (IMU); camera; monitor; as well as programming in the memory, wherein execution of the programming by the processor configures the portable electronic device to perform functions including: capturing frames of video data with the camera; detecting the IoT product at its location in the physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; detecting, using the IMU, an orientation of the device relative to the physical environment; presenting a virtual rotation interface on the display at a viewport position based on the IoT product position and the portable electronic device position, wherein the virtual rotation interface includes a slider that is virtually presented at a slider position based on the detected device orientation, and wherein the slider position corresponds to an IoT action; and A control signal is sent to the IoT product based on the detected device orientation, wherein the control signal includes instructions for the IoT product to perform the IoT action.

10. The portable electronic device according to claim 9, wherein: Execution of the programming also configures the portable electronic device to perform the following functions: Detecting the execution of the IoT action by the IoT product according to the instruction.

11. The portable electronic device according to claim 9, wherein: The IoT product includes a controllable feature characterized by a current value, and wherein execution of the programming further configures the portable electronic device to perform the following functions: When the portable electronic device is located within a threshold proximity of the IoT product, presenting the virtual rotation interface; detecting a starting device orientation when the portable electronic device position is first detected within the threshold proximity; associating the starting device orientation with a current value of the controllable characteristic; detecting a final device orientation when the portable electronic device position is no longer detected within the threshold proximity; and The final device orientation is associated with the IoT action.

12. The portable electronic device according to claim 9, wherein: Execution of the programming also configures the portable electronic device to perform the following functions: A final device orientation is detected in response to the user input.

13. The portable electronic device according to claim 9, wherein: Execution of the programming also configures the portable electronic device to perform the following functions: presenting the slider relative to a viewport; presenting additional sliders virtually rendered at additional slider positions relative to the additional viewport, wherein the additional slider positions correspond to additional IoT actions; In response to user input, associating the detected device orientation with at least one of the slider and the additional slider; and At least one of the slider and the additional slider is presented based on the association and the detected device orientation.

14. The portable electronic device according to claim 9, wherein: The IoT product is associated with a product API, the product API comprising a product type and controllable features associated with a variable value range, and wherein execution of the programming further configures the portable electronic device to perform the following functions: establishing a virtual ruler relative to the orientation of the device, the virtual ruler extending angularly clockwise from a low end to a high end; calibrating the virtual ruler using the variable value range; and The slider is rendered relative to a viewport that is virtually rendered according to the calibrated virtual ruler.

15. The portable electronic device according to claim 9, wherein: Execution of the programming also configures the portable electronic device to perform the following functions: estimating a device rotation plane using the IMU; and The virtual rotation interface is presented according to the estimated device rotation plane.

16. The portable electronic device according to claim 9, wherein: Execution of the programming also configures the portable electronic device to perform the following functions: Detecting execution of the IoT action by the IoT product, wherein the execution includes selecting an IoT action based on the detected device orientation.

17. A non-transitory computer readable medium storing program code, the program code being operable when executed to cause a processor of a portable electronic device to perform the following steps: capturing frames of video data with a camera coupled to the portable electronic device; detecting an Internet of Things (IoT) product at a location of the IoT product in a physical environment using the captured video data frames; Using the captured video data frames to determine a location of the portable electronic device relative to the location of the IoT product; detecting an orientation of the device relative to the physical environment using an inertial measurement unit (IMU) coupled to the portable electronic device; presenting a virtual rotation interface at a viewport position on a display coupled to the portable electronic device based on the IoT product position and the portable electronic device position, wherein the virtual rotation interface includes a slider that is virtually presented at a slider position based on the detected device orientation, and wherein the slider position corresponds to an IoT action; and A control signal is sent to the IoT product based on the detected device orientation, wherein the control signal includes instructions for the IoT product to perform the IoT action.

18. The non-transitory computer readable medium storing program code according to claim 17, wherein: The program code is operable when executed to cause the processor to perform the following further steps: detecting the IoT action performed by the IoT product according to the instruction; When the portable electronic device is located within a threshold proximity of the IoT product, presenting the virtual rotation interface; detecting a starting device orientation when the portable electronic device position is first detected within the threshold proximity; associating the starting device orientation with a current value associated with a controllable characteristic; detecting a final device orientation when the portable electronic device position is no longer detected within the threshold proximity; detecting a final device orientation in response to a user input; as well as The final device orientation is associated with the IoT action.

19. The non-transitory computer readable medium storing program code according to claim 17, wherein: The program code is operable when executed to cause the processor to perform the following further steps: rendering the slider relative to a viewport; presenting additional sliders virtually rendered at additional slider positions relative to the additional viewport, wherein the additional slider positions correspond to additional IoT actions; In response to user input, associating the detected device orientation with at least one of the slider and the additional slider; and At least one of the slider and the additional slider is presented based on the association and the detected device orientation.

20. The non-transitory computer readable medium storing program code according to claim 17, wherein: The program code is operable when executed to cause the processor to perform the following further steps: establishing a virtual ruler relative to the orientation of the device, the virtual ruler extending angularly clockwise from a low end to a high end; calibrating the virtual ruler with a variable value range associated with a controllable characteristic of the IoT product; as well as The slider is rendered relative to a viewport that is virtually rendered according to the calibrated virtual ruler.

Images