Standardized AR interface for IOT devices

Abstract

An augmented reality (AR) device interacts with an Internet of Things (IoT) device by receiving IoT device information in a standardized mode from the IoT device. The IoT device information includes device input and available output information for the IoT device. A predetermined AR user interface widget to present for the IoT device is determined from the received IoT device information. The predetermined AR user interface widget translates user input to the AR device into device input for the IoT device. Upon selection of the IoT device by the AR device, a predetermined AR user interface widget for the selected IoT device is provided to the AR device as a cover on a display of the AR device. A device input received in response to a user interaction with the AR user interface widget is provided to the IoT device in an input type desired by the IoT device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 841,365, filed on June 15, 2022, and is related to U.S. application serial number 17 / 841,397, filed on June 15, 2022, entitled “AR SYSTEM FOR PROVIDING INTERACTIVE EXPERIENCES INSMART SPACES,” the contents of both of which are fully incorporated herein by reference. Technical Field

[0003] The present subject matter relates to systems and methods for enabling users of camera-based augmented reality (AR) systems to interact with real-world Internet of Things (IoT) devices, and more particularly, to systems and methods for providing a standardized user interface that enables AR cameras to interact with many common IoT devices. Background Art

[0004] The so-called "Internet of Things" or "IoT" is a network of physical objects embedded with sensors, software, and other technologies for enabling connectivity and data exchange with other devices via the Internet. For example, IoT devices are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided that act as smart home hubs for connecting different smart home products. IoT devices have also been used for many other applications. Application layer protocols and supporting frameworks for implementing such IoT applications have been provided. Artificial intelligence is also combined with IoT infrastructure to enable more efficient IoT operations, improve human-machine interactions, and enhance data management and analysis.

[0005] An increasing number of IoT devices have become available to consumers in the home. These devices range from smart red, green, and blue (RGB) light bulbs to automatic plant watering systems. At the same time, the adoption of AR technology is growing, for example in applications such as Currently, augmented reality (AR) is mainly used to enhance photos and videos, and is also used in interactive applications (e.g. However, there has been little work on the topic of using AR as an interface for connected IoT devices, which are currently a diverse mix of two-dimensional (2D) phones or web-based applications, often using different user experience designs to accomplish the same basic tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The features of the various embodiments disclosed will be readily understood from the detailed description made below with reference to the attached drawings. Reference numerals are used with each element in the specification and throughout the multiple views of the drawings. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements, with the additional lower case letters referring to the specific element.

[0007] The various elements shown in the drawings are not drawn to scale unless otherwise specified. The sizes of the various elements may be enlarged or reduced for clarity. The various figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0008] Figure 1 is a general block diagram of a system for connecting an augmented reality (AR) camera device to an Internet of Things (IoT) device;

[0009] Figure 2 In the sample configuration Figure 1 A block diagram of the system's components and information flow;

[0010] Figure 3 is a block diagram of a component layer of an AR application built on an AR user interface widget in a sample configuration, the AR user interface widget being exposed as an application programming interface (API);

[0011] Figure 4 is a diagram depicting a selectable numeric range widget in an example application for controlling a smart thermostat using sample gesture input;

[0012] Figure 5 is a diagram depicting a numeric range input formed by mapping distances between hand pinches to values ​​in a sample gesture input;

[0013] Figure 6 is a diagram depicting controlling a mobile robot along a two-dimensional path using tracked finger coordinates provided via a sample gesture input;

[0014] Figure 7 is a diagram depicting a hue, saturation, value (HSV) color picker for a smart light controlled using gestures;

[0015] Figure 8 is a flow chart of a method for interacting an AR camera device with an IoT device via a standardized AR interface in a sample configuration;

[0016] Fig. 9 shows a sample configuration of a mobile phone suitable for operating as an AR camera device in a sample configuration; and

[0017] Fig.10 A sample configuration of a computer system suitable for implementing the back-end server systems and methods described herein is shown. DETAILED DESCRIPTION

[0018] Although augmented reality (AR) technology is becoming more and more common, control of IoT devices remains largely in the realm of two-dimensional web or phone-based user interfaces, with very diverse design choices. Systems and methods described herein are directed to utilizing AR camera interfaces to affect real-world Internet of Things (IoT) objects by providing a standardized AR user interface for many common IoT devices.

[0019] The systems and methods described herein solve the technical problems of mapping user interface parameters of IoT devices to a set of standardized AR user interface widgets and designing AR user interface widgets to enable input of certain variable types used by common IoT devices. As used herein, a "widget" is a component of an application or interface that enables a user to perform a function or access a service.

[0020] To address these technical issues, the standardized AR user interface described herein maps IoT device input variables to appropriate AR user interface widgets. Many common IoT device input variable types and their corresponding mappings to AR gesture input interfaces are described for use by the system. By utilizing the camera on various messaging or AR platforms, such as those available from Snap Inc. of Santa Monica, California, or from SPECTACLES TM , users can create fun and immersive ways to interact with IoT devices through standardized AR interfaces.

[0021] The systems and methods described herein interface augmented reality (AR) devices with Internet of Things (IoT) devices by receiving IoT device information in a standardized schema. The IoT device information represents at least one device input and available output information for the IoT device. A predetermined AR user interface widget to be presented for the IoT device is determined from the received IoT device information. For example, IoT device inputs and outputs may be mapped to available standardized AR user interface widgets to determine which standardized AR user interface widgets can interpret IoT device inputs and outputs. The predetermined AR user interface widget converts user input (e.g., gesture input) to the AR device into at least one device input for the IoT device. After the IoT device is selected by the AR device, the predetermined AR user interface widget for the selected IoT device is provided to the AR device as an overlay on the display of the AR device. In response to user interaction with a predetermined AR user interface widget displayed by the AR device, the AR device input is converted into a device input type expected by the IoT device and provided to the IoT device.

[0022] The system described herein may include an AR device that interacts with or includes a registration module that receives IoT device information in a standardized format. The AR device may also interact with or include a communication module that facilitates communication of input data from the AR device to the IoT device and status data from the IoT device to the AR device. The AR widget user interface may determine a predetermined AR user interface widget to be presented for the IoT device from the received IoT device information. The predetermined AR user interface widget for the IoT device may be provided to the AR device as an overlay on the display of the AR device. The predetermined AR user interface widget is suitable for converting user input from an input device of the AR device into at least one device input for the IoT device. During use, the converted user input captured in response to user interaction with the AR user interface widget displayed by the AR device is provided to the IoT device to control the operation of the IoT device.

[0023] The AR widget user interface can compare the received IoT device information with the available standardized AR user interface widgets to determine at least one standardized AR user interface widget that can interpret the IoT device input and output, and select a standardized AR user interface widget from the available standardized AR user interface widgets, which converts the user input to the AR device into the following form of IoT device input according to the form of the device input of the IoT device: for example, Boolean, range value, two-dimensional relative value, two-dimensional absolute value, two-dimensional path, three-dimensional path, red, green, blue (RGB) color input, time, or text.

[0024] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products, which illustrate the examples set forth in the present disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without such details. The aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in a variety of ways. The terms and nomenclature used herein are only used to describe the purpose of specific aspects and are not intended to be restrictive. In general, known instruction instances, protocols, structures, and techniques do not need to be shown in detail.

[0025] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links or the like, by which an electrical or magnetic signal generated or supplied by one system element is communicated to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communications media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0026] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art after reviewing the following and the accompanying drawings, or may be learned by production and operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, means and combinations particularly pointed out in the appended claims.

[0027] Several features of AR make it attractive for controlling IoT devices. Currently, user interfaces for IoT devices are quite diverse, with devices from different brands relying on different applications (e.g., on mobile devices) to control IoT devices. To prevent the proliferation of similarly diverse applications in the AR (or even virtual reality (VR)) space, a better approach would involve IoT devices specifying their inputs in a common way, and, as suggested in this article, using application programming interfaces (APIs) to provide input user interfaces for IoT devices in a standardized way.

[0028] AR devices are spatially aware, and IoT devices are often distributed in the user's space. These characteristics make it possible to select and manipulate a specific IoT device in AR by pointing to its location in space, rather than selecting it from a (potentially long) list of available devices in two dimensions. IoT devices may have an expanded set of inputs or multi-dimensional inputs. Spatial interactions using AR have the potential to extend beyond the screen of a mobile device while still being understandable and navigable by the user. In addition, spatial input modalities such as gestures are useful for inputting multi-dimensional variables (e.g., RGB color selection) compared to the options provided by a 2D touchscreen.

[0029] To take advantage of these features, this article provides systems and methods for supporting spatial input for IoT devices in AR applications using universal or "standardized" AR user interface widgets. In addition, a set of spatial AR user interface widgets for universal input for IoT devices is described.

[0030] Now refer to the attached Figures 1 to 10 An example is illustrated in and discussed below.

[0031] In its most basic form, the system includes one or more IoT devices, one or more AR devices, a backend server including a registration module and a dispatch (communication) module, and an AR widget user interface for the AR devices. Figure 1 1 is a general block diagram of a system 100 for interacting an augmented reality (AR) camera device with an Internet of Things (IoT) device in a sample configuration. As shown, a plurality of AR-enabled camera devices 1101 to 110 N (such as suitable for including messaging applications (such as those available from Snap Inc. of Santa Monica, California ) of the Snapchat camera) respectively include AR displays 1201 to 1202 for displaying AR objects N The AR camera device 110 may include a smartphone, a laptop computer, a desktop computer, an electronic eyewear device, a head up display, or any other computing device capable of communicating via the Internet. N It may further be adapted to include software 1301 to 130 N To support features such as hand and body recognition and tracking, marker tracking in AR space, and the like. Such features are readily available in messaging applications, such as the aforementioned Messaging apps.

[0032] In the sample embodiment, the corresponding AR camera devices 1101 to 110 NA common AR widget user interface 140 may be shared. It will be appreciated that in a sample configuration, the AR user interface widget and related object recognition services may also be provided by each AR camera device 1101 to 110 N Software running on 1301 to 130 N To perform, and in a sample configuration, a common AR widget user interface 140 may be provided separately for each AR camera device 110. As will be described in the example below, the AR camera device 110 may present an AR user interface widget to the AR display 120 of the AR camera device 110. The user may make a selection using gesture input recognized by the software 130 to generate a device input representing the user's selection. The device input is then provided to the IoT device 160 to control the physical action 180 of the IoT device 160.

[0033] If you will refer to Figure 2 As explained in more detail, the backend server 150 acts as a bridge between the AR camera device 110 and the physical IoT device 160. The backend server 150 can also send commands to the IoT device 160 to act in any number of ways supported by the corresponding IoT device 160. The IoT device 160 can be controlled to respond to the AR camera device 1101 to 110 N The user interacts with the AR camera device 1101 to 110 and performs the desired physical action 180 and provides the sensor output to initiate the communication with the AR camera device 1101 to 110 N One-way or two-way interaction. Such two-way interaction may include transmitting status information from IoT device 160 to the corresponding AR camera device 110. Backend server 150 may also respond to context information provided by other context servers 170. Such context information may include time of day, weather conditions, global positioning system (GPS) data, and any other information available from third-party servers.

[0034] Figure 2 In the sample configuration Figure 1 1 is a block diagram of the components and information flow of the system 100. As shown, the system 100 includes an IoT device 160, an AR device 110, a backend server 150, and an AR widget user interface 140. The system 100 provides a standardized user interface so that a user of the AR device 110 can interact with one or more IoT devices 160 in a space.

[0035] IoT device 160 may be any available connected smart device known in the art. For example, IoT device 160 may include common household devices that a standard end user may encounter, such as RGB smart lights, thermostats, smart speakers, smart switches, and the like. The examples provided herein do not provide interfaces to devices found in commercial spaces, such as devices that may be used to create a specific interactive experience for consumers. However, those skilled in the art will appreciate that as long as such devices are connected via Or other communication channels that enable Internet access provide connections to the AR device 110 and the backend server 150, and the device also provides active or passive positioning in the environment that can be received by the AR device 110, such IoT devices 160 can be suitable for working in the system described in this article. Active positioning can be achieved through radiofrequency (RF) beacons, ultrasonic waves, visible light signals, or via a global positioning system (GPS) for outdoor applications. On the other hand, passive positioning can be achieved by saving the fixed position of the IoT device 160 in the memory of the backend server 150 or the AR device 110 or through a visual code near the IoT device 160 for selection or other trustees that the AR device 110 can recognize. The AR device 110 and the IoT device 160 can also perform limited duplex communication to convert user input into the AR device 110 into a display of an AR user interface widget suitable for the IoT device 160.

[0036] Since the system 100 provides a standardized interface for the AR device 110 to interact with the IoT device 160, a suitable form factor is desirable for the AR device 110, such as a smartphone, a tablet, or glasses. In a sample configuration, the AR device 110 is connected to the IoT device 160 via a communication channel that enables Internet access (e.g., The AR device 110 provides a connection to the backend server 150 or directly to the IoT device 160, and also provides inside-out or outside-in positioning capabilities. In one example, the positioning capability of the AR device 110 generates a relative pose (position, orientation) between the AR device 110 and any IoT device 160 to be controlled via the AR user interface widget in the sample configuration. In addition, in addition to standard input modes (e.g., touch screen, buttons, trackpad, voice), the AR device 110 can support spatial input such as gesture or body posture tracking.

[0037] Figure 33 is a block diagram 300 of a widget layer including an AR application 310 in a sample configuration that is built to run on an AR widget user interface 320 that is exposed as an application programming interface (API) on the processor of the AR device 110. The API 320 in turn leverages the capabilities of the AR device operating system 330 to communicate with the backend server 150 and access other information, such as the position and orientation of the AR device 110 in space, and user input tracking events.

[0038] like Figure 2 As shown, the backend server 150 includes two main modules, a registration module 152 and a dispatch (communication) module 154. The registration module 152 is used by the IoT devices 160 to register their device input and available output information. The IoT devices 160 can broadcast their presence to the backend server 150 when powered on to initiate the registration process. Conversely, the backend server 150 can periodically poll for new IoT devices 160. The inputs and outputs of the AR user interface widgets to be used for the corresponding IoT devices 160 can be specified to define the AR user interface widgets using a standardized schema, such as the following example JSON file, which shows a minimum configuration for controlling a smart light:

[0039] {

[0040] "deviceUUID":"bd1b12ffcdf38f4e80b2ccd7d3660548",

[0041]

[0042] Similarly, the AR devices 110 may register with the backend servers 150 using the registration module 152 and signal that they are ready to receive updates from the IoT devices 160 and also send AR updates to those IoT devices 160. In addition, the AR devices 110 may send their current location to the backend servers 150, which may coordinate certain updates between the IoT devices 160 and the AR devices 110 (e.g., start sending a certain state to the AR devices 110 or activate a certain input AR widget user interface (UI) 140 on the AR devices 110). However, some parts of such logic may also run locally on the AR devices 110.

[0043] The dispatch module 154 facilitates the communication of input data from the AR device 110 to the IoT device 160 and status data (e.g., current temperature) from the IoT device 160 to the AR device 110. In some configurations, the IoT device 160 can push a preferred AR overlay, which is communicated to the AR device by the dispatch module 154.

[0044] It is understood that the backend server 150 can be run on a separate machine, or as a further API or system layer on the AR device operating system 330 of the AR device 110 itself, such as Figure 3 shown.

[0045] In a sample configuration, the AR widget UI 140 selects and makes available to the user an AR user interface widget that accepts input and output for interacting with the corresponding IoT device 110. The AR widget UI 140 may serve as an AR device operating system 330 ( Figure 3 ) layered between widget API 320.

[0046] The AR widget UI 140 determines the type of AR user interface widget to be presented for a given IoT device 160 that can interpret the input and output of the given IoT device 160 based on the input and output information for the given IoT device 160 provided by the registration module 152. For example, the IoT device input and available output information of the corresponding IoT device 160 can be compared or mapped to the input and available output information for the available AR user interface widgets to determine which AR user interface widgets can interpret the IoT device input and output. The dispatch module 154 sends the input captured by the AR user interface widget to the appropriate IoT device 160 or obtains and / or receives updates of state data from the given IoT device 160 for visualization on the AR device 110. The AR widget UI 140 transforms the original user input into input suitable for the IoT device 160 to control the state data transmitted from the IoT device 160 and transforms it into a user-understandable representation and visualization for display. For example, the AR widget UI 140 may map the AR device 110 to a corresponding AR user interface widget for a given IoT device 160. The AR widget UI 140 may also draw graphics and specify the raw gesture data transformation required to control a particular AR user interface widget. The raw gesture may be converted to a value as follows: Figures 4 to 7 In addition, the status data from the IoT device 160 can be provided to the AR device 110 for display. Therefore, the AR device 110 can receive feedback from the IoT device 160 via two-way communication for display to the user.

[0047] In the system 100, various AR interfaces are provided for common input variables of IoT devices 160. Multiple AR user interface widgets may also be used. These AR user interface widgets are designed to be generic and standardized to cover many common input data types of IoT devices 160. AR user interface widgets are designed for spatially aware AR devices and may be activated, for example, when a user arrives in close proximity to an IoT device 160 that the user wishes to control. Conversely, an AR user interface widget may be activated when a user is oriented to face an IoT device 160 to be controlled, when an IoT device 160 is selected by gesture input, or when an IoT device 160 is selected by a multimodal voice command.

[0048] Table 1 below provides a non-exhaustive list summarizing some common input variable data types for controlling IoT devices 160 and their possible implementations as standardized AR user interface widgets (W1-W6), which provide mechanisms for manipulating standard input variable data types. Standardized AR user interface widgets can have default values ​​and can be customized based on user input to cover known IoT device input / output combinations. Therefore, the available predetermined AR user interface widgets for a particular IoT device can include standardized AR user interface widgets and AR user interface widgets that have been customized by the user.

[0049] Table 1

[0050]

[0051]

[0052] Possible AR user interface widget configurations for the standardized AR user interface widgets W1 - W4 in Table 1 will be explained below.

[0053] In the momentary or selectable state toggle widget (widget W1), a Boolean input can represent a true / false value that is sent to the IoT device 160. A Boolean input can be a momentary signal that is only valid while the user "holds" the input (e.g., a doorbell buzzer) or a state-holding switch (e.g., an on / off toggle switch). A possible way to implement such input using a spatial AR interface is by tracking whether the user's hand or finger crosses a threshold of the 2D plane or enters the interior of a 3D volume. A 3D gesture classifier can also be used to transmit a momentary value, a true or false value, or a switch between any of the aforementioned logical states.

[0054] In a selectable numeric range widget (widget W2), the selectable numeric range has lower and upper limits and a selection granularity, i.e., the interval between one selectable value and the previous or next selectable value. The selectable numeric range can typically be either integer or floating point, although the floating point selection accuracy is ultimately limited by the resolution or tracking method of the input device or the dexterity of the user.

[0055] Figure 4 is a diagram depicting a selectable numeric range widget 400 in an example application for controlling a smart thermostat 410 using example gesture input. Note that the center of the range is set to the previous target temperature of the thermostat. The AR user interface widget 400 can present a dynamic interface on the display of the user's AR device 110 based on proximity to a specific IoT device 160 (such as the smart thermostat 410). The user can gesture as indicated relative to the AR user interface widget 400 to change the set point of the smart thermostat 410. The AR device 110 recognizes the user selection based on the user's gesture and provides the selection as a device input to the backend server 150, which in turn provides the device input to the specific IoT device 160, in this case the smart thermostat 410.

[0056] Selectable numeric ranges can be selected using only gestures. Numeric range input can be specified by mapping the distance between two hand pinches to a value. For example, Figure 5 is a diagram depicting a numeric range input formed by mapping the distance between pinches 510 and 520 to a value 530 in a sample gesture input.

[0057] Other possible ways to implement range selection input using a spatial AR interface include representing the input as a slider along a numbered scale, or by representing such a scale as a circular dial. The slider handle or dial can be manipulated via hand and finger gestures, such as by "pinch and drag". More complex ways to implement range selection input can use interpretation of certain parameters of spatial gestures, for example, by mapping the distance between the user's hands to a range of input values, such as Figure 5 shown.

[0058] In the measurement of the Relative 2D Input widget (Widget W3a), the Relative 2D Input measures the difference of the input value from a set center point or starting value. This is similar to a joystick or track point input that tracks relative or absolute values ​​along a track path. The former measures the degree of rotation away from the center in two axes, while the latter measures the amount of force applied to the track point in two dimensions. As an example, the user can use Figure 5The user may use a pinch gesture to raise or lower the volume of the IoT device 160 by moving the user's hands farther away or closer, respectively. As an example, a value (such as volume) may pop up on the display of the user's AR device 110 as the user approaches the IoT device 160. The user may use gestures to adjust the value for the IoT device 160 by, for example, moving the user's hands closer together or farther apart (as just described). The AR device 110 recognizes the user selection based on the user's gestures and provides the selection as a device input to the backend server 150, which in turn provides the device input to the specific IoT device 160 that responded to the numeric range input.

[0059] Relative 2D input can be implemented on an AR device 110 with spatial input in a variety of ways. A first possibility is by measuring the (2D) displacement from the starting point during a drag gesture. A second possibility is to set a certain movement speed threshold and map the movement speed and direction at the threshold to a relative 2D input. An alternative approach may use an untethered force feedback controller or force transducers and sensors mounted on the user's hands and / or fingers. The user's own motion relative to a specified point in space can also be used as a relative 2D input.

[0060] In a spatial AR interface, absolute 2D input can be achieved by selecting a location on a 2D plane via ray intersection, moving a proxy object on a 2D plane, or by direct spatial gestures. Alternative methods may include creating a virtual projection of a "UI map" onto the floor of the user's environment and using the user's current coordinates on the projection as absolute 2D input. As in other examples, the AR device 110 recognizes the user selection based on the user's gestures and provides the selection as device input to the backend server 150, which in turn provides the device input to the IoT device 160 that expects absolute 2D input.

[0061] In the absolute 2D position input widget (widget W3b), the absolute 2D position input maps 2D values ​​from the user input coordinate system to the coordinate system used by the IoT device 160 to be directly controlled. For example, this type of input allows the user to select a real-world location from a "mini map" in the user interface. The coordinates selected by the user in the user interface are recognized as device input by the user's AR device and provided to the backend server 150, which in turn maps the device input from the user interface coordinates to real-world coordinates provided to the IoT device 160.

[0062] For the widget for inputting a sequence of coordinates (widget W3c), Figure 6A diagram depicting the control of a mobile robot 600 along a two-dimensional path using tracked finger coordinates provided via a sample gesture input 610 is shown. The tracked finger coordinates are projected onto the floor of the user environment in a perspective corrected manner. A 2D or 3D path may be defined by inputting a sequence of coordinates. A path may be used, for example, to define the motion of the mobile robot 600 in a certain area. In the 2D case, this may be accomplished by using ray intersections on a plane, dragging a proxy object on a plane, or by using, for example, Figure 6 The illustrated spatial gesture 610 inputs a path via spatial AR. To create a 3D path, a proxy object may be dragged in the 3D volume, or a direct 3D gesture may be performed to specify the desired path. A 2D path on the ground may be input directly in space by drawing on the floor of the user's environment, such as projecting the user's finger motion onto the floor of the environment, such as Figure 6 As shown. Alternatively, a path can be specified by saying a trigger word and then pointing to the desired destination. For each case, the AR device 110 recognizes the user selection based on the user's gesture and provides the selection as device input to the backend server 150, which in turn provides the device input to the specific IoT device 160 as, for example, the coordinates of the desired path.

[0063] To implement a color picker widget (widget W4), the color picker widget allows the user to select a specific color as input, for example, to change the color of a smart light bulb. In many applications, colors are represented as RGB (red, green, blue) or HSV (hue, saturation, value) triplets, although there are many other color space representations to choose from. The color picker can present an array of pre-selected colors or allow the user to select the desired color in a "free-form" manner, for example, selecting from a color wheel. HSV is commonly used for free-form color pickers.

[0064] Figure 7 is a diagram depicting a spatial AR user interface of a 3D HSV color picker 700 for a smart lamp 710 controlled by an AR device 110 using gestures. In a sample configuration, the HSV color picker 700 appears on the display of the AR camera device 110 when the user enters the vicinity of the lamp 710 to interact with the lamp 710. Figure 7As shown, hue can be adjusted by a user interacting with the HSV color picker 700 by moving her finger or hand up or down in the color circle segment (as shown at 720), saturation can be adjusted by a user interacting with the HSV color picker 700 by moving her finger or hand closer to the light fixture 710 or further away from the light fixture 710 (as shown at 730), and brightness value can be adjusted by a user interacting with the HSV color picker 700 by moving her finger or hand in a Z direction orthogonal to the hue axis and the saturation axis (e.g., at point 740). The AR device 110 recognizes the user selection based on the gesture of the user interacting with the color picker 700 and provides the selection as device input to the backend server 150, which in turn provides the device input to a specific IoT device 160, in this case, the light 710.

[0065] It will be appreciated that fixed palette selections can be implemented using Boolean widgets. Spatial AR user interfaces can also be implemented as Figure 7 3D HSV color picker shown. As shown, the user can select hue and saturation by selecting points on the color circle (X-axis and Y-axis), such as through finger gestures, and select brightness values ​​by moving their hands forward or backward along the Z-axis. An alternative method for color selection in an AR user interface can use a more direct interaction between the user and the physical environment. For example, by using the image and touch position tracking of the camera of the AR device 110, the user can select a color from a real-world object by touching the real-world object. The AR user interface will then assign the color at the touch location to the selected object. Similar color selection techniques can also be used for virtual objects in an AR environment.

[0066] Figure 8 800 is a flow chart of a method 800 for interacting an AR camera device 110 with an IoT device 160 via a standard AR interface in a sample configuration. The method may be implemented by the server 150 alone ( Figure 2 ), implemented by an AR device 110 that implements server functions, or implemented by a combination of a server 150 and one or more AR devices 110.

[0067] like Figure 8As shown, at 810, each IoT device 160 registers with the server 150 by providing its device input (see, e.g., Table 1) and available output information using a standardized schema. At 820, the server 150 determines the type of AR user interface widget to be presented for each IoT device 160 from the registration data provided by the IoT device 160. For example, the IoT device input and output can be mapped to available standardized AR user interface widgets to determine which standardized AR user interface widgets can interpret the IoT device input and output. This information can be stored in a table on the server 150 or in the AR device 110. At 830, each AR device 110 is also registered. At 830, location information of each AR device 110 can also be obtained.

[0068] During operation, at 840 the server 150 receives a selection of an IoT device 160 to be controlled by the AR device 110. As noted above, the selection may be based on the proximity of the AR device 110 to the IoT device 160, verbal input from the user, orientation by the AR device 110 toward the IoT device 160, and the like. At 850, an AR user interface widget for the selected IoT device 160 is provided to the AR device as an overlay on the display of the AR device 110. The AR user interface widget may appear automatically or may be selected by the user. Once the AR user interface widget has been displayed to the user, input captured by the AR user interface widget (e.g., user gestures) may be converted by the AR user interface widget into an IoT device input in a format suitable for the IoT device 160 and provided to the IoT device 160 to control the operation of the IoT device 160. Conversely, at 860 state data from the IoT device 160 may be provided to the display of the AR device 110. Thus, the AR device 110 may recognize a user selection based on a user gesture relative to a displayed AR user interface widget and provide the selection as a device input to the backend server 150 , which in turn provides the device input to a specific IoT device 160 .

[0069] For each registered IoT device 160 and each AR device 110, the selection of the IoT device 160 to be controlled, the provision of an AR user interface widget for the selected IoT device 160, and the interaction with the AR user interface widget (e.g., through user gestures) to generate device input to control the IoT device 160, and the receipt of status data from the IoT device 160 can be repeated.

[0070] In one use case example, a user wears the AR device 110 and enters his home office in the morning to set it up for the day. He approaches his IoT-enabled desk lamp, which is currently off. Based on the proximity of the user's AR device 110 to the IoT-enabled desk lamp, an "on / off" button AR user interface widget (widget W1) appears in his field of view next to the desk lamp as an AR overlay on his AR device 110. He extends the index finger of his right hand and taps the AR user interface widget. The desk lamp turns on. However, it is still set to a "warm light" yellow / brown color based on the "warm light" yellow / brown color that the user relaxed the previous evening. To change the color of the light, the user moves his hand closer to the desk lamp's bulb and an AR color picker (widget W4) appears. The user uses the displayed picker and the like to select a color. Figure 7 The pinch gesture shown selects a bluer and brighter light tone. The room feels a bit cold, so the user moves closer to an air duct in the room. Based on the proximity of the user's AR device 110 to the air duct, a virtual thermostat control (widget W2) for an IoT-enabled thermostat appears on the AR overlay on his AR device 110. The user uses a double pinch gesture and Figure 5 As shown, the user moves his hands apart to raise the preset temperature of the room. Now that his work environment is set (and without physically touching anything in the room), the user can begin to work.

[0071] System Configuration

[0072] Fig. 9 A sample configuration of a mobile device 900 suitable for interacting the AR camera device 110 with the IoT device 160 in a sample configuration is shown. In particular, Fig. 9 900 is a high-level functional block diagram of an example mobile device 900 that a user may use as an AR camera device 110 as described herein. The mobile device 900 may include a flash memory 905 that stores programming to be executed by the CPU 910 to perform all or a subset of the functions described herein. For example, the flash memory may store AR camera software 915 and optionally include AR processing software 920 for execution by the CPU 910 to enable a user of the mobile device 900 to display and interact with AR objects presented by the AR camera device 110 as described herein. The mobile device 900 may further include a camera 975 that includes one or more visible light cameras (a first visible light camera and a second visible light camera with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 905 may further include a plurality of images or videos generated via the camera 975.

[0073] The mobile device 900 may further include an image display 930, a mobile display driver 935 for controlling the image display 930, and a display controller 940. Fig. 9 In the example of , the image display 930 may include a user input layer 945 (e.g., a touch screen) layered on top of or otherwise integrated into the screen used by the image display 930. Examples of touch screen type mobile devices that may be used include, but are not limited to, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, Fig. 9 Thus, a block diagram is provided showing an example mobile device 900 having a user interface including a touch screen input layer 945 for receiving input (via touch, multi-touch or gestures, etc., by hand, stylus or other implement) and an image display 930 for displaying content.

[0074] like Fig. 9 As shown, the mobile device 900 includes at least one digital transceiver (XCVR) 950, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 900 may also include additional digital or analog transceivers, such as a short-range transceiver (XCVR) 955 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, or For example, the short-range XCVR 955 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard protocols for communications implemented in wireless local area networks, such as the IEEE 802.11 standard. One of the standards.

[0075] In order to generate location coordinates for locating the mobile device 900, the mobile device 900 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 900 may utilize either or both of the short-range XCVR 955 and the WWAN XCVR 950 to generate location coordinates for positioning. For example, a cellular network, or based on The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the mobile device 900 via the XCVR 950, 955 through one or more network connections.

[0076] The transceivers 950, 955 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 950 include, but are not limited to, transceivers configured to operate according to code division multiple access (CDMA) and third generation partnership project (3GPP) network technologies, including, for example, but not limited to, 3GPP type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". The transceiver may also be incorporated into broadband cellular network technology known as "5G". For example, transceivers 950, 955 provide two-way wireless communication of information including digitized audio signals, still images and video signals, web page information for display and input associated with the web page, and various types of mobile message communications to / from the mobile device 900.

[0077] The mobile device 900 may further include a microprocessor that functions as a central processing unit (CPU) 910. A processor is a circuit having elements that are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components may be used, examples utilize components that form a programmable CPU. The microprocessor, for example, includes one or more integrated circuit (IC) chips that integrate electronic components to perform the functions of the CPU 910. The CPU 910, for example, may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form a CPU 910 or processor hardware in a smartphone, a laptop computer, and a tablet computer.

[0078] The CPU 910 acts as a programmable host controller for the mobile device 900 by configuring the mobile device 900 to perform various operations, for example, according to instructions or programming executable by the CPU 910. For example, such operations may include various general operations of the mobile device 900, as well as operations related to programming for a messaging application and an AR camera application on the mobile device 900. Although the processor may be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing programming.

[0079] The mobile device 900 also includes a memory or storage system for storing programming and data. Fig. 9In the example shown in , the memory system may include flash memory 905, random-access memory (RAM) 960, and other memory components 965 as needed. RAM 960 may be used as short-term storage for instructions and data being processed by CPU 910, for example, as working data processing memory. Flash memory 905 generally provides longer-term storage.

[0080] Thus, in the example of mobile device 900, flash memory 905 may be used to store programming or instructions for execution by CPU 910. Depending on the type of device, mobile device 900 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPAD devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or the like.

[0081] Finally, the mobile device 900 may include an audio transceiver 970 that may receive audio signals from the environment via a microphone (not shown) and provide audio output via a speaker (not shown). The audio signals may be coupled with the video signals and other messages by a messaging application or social media application implemented on the mobile device 900. In addition, the audio signals may include verbal commands from the user regarding the IoT device (e.g., "turn on the lights").

[0082] The technology described herein can also be used with one or more computer systems described herein or with one or more other systems. For example, the various processes described herein can be implemented with hardware or software, or a combination of the two. For example, at least one of the processors, memories, storage, output devices, input devices, or communication connections discussed below can each be at least a portion of one or more hardware components. A dedicated hardware logic component can be constructed to implement at least a portion of one or more of the technologies described herein. For example and without limitation, such hardware logic components can include field programmable gate arrays (FPGAs), program-specific integrated circuits (ASICs), program-specific standard products (ASSPs), systems-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. The applications of the devices and systems that can include various aspects can widely include various electronic and computer systems. The technology can be implemented using two or more specific interconnected hardware modules or devices with related control and data signals, which can communicate between modules and through modules, or as part of a specific application integrated circuit. In addition, the techniques described herein can be implemented by a software program executable by a computer system. As an example, implementations can include distributed processing, component / object distributed processing, and parallel processing. Moreover, virtual computer system processing can be constructed to implement one or more techniques or functions, as described herein.

[0083] Fig.10 A sample configuration of a computer system suitable for implementing the systems and methods described herein is shown. In particular, Fig.10A block diagram of an example of a machine 1000 is shown, on which one or more configurations of an AR camera device 110, an AR widget user interface 140, a backend server 150, and a context server 170 can be implemented. In alternative configurations, the machine 1000 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, the machine 1000 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. In a sample configuration, the machine 1000 can be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web device, a server, a network router, a switch or a bridge, or any machine capable of executing instructions (continuously or otherwise) specifying the actions to be taken by the machine. For example, the machine 1000 can be used as a workstation, a front-end server, or a back-end server of a communication system. The machine 1000 can implement the methods described herein by running software for implementing features as described herein to enable the AR device 110 to control the IoT device 160. In addition, although only a single machine 1000 is illustrated, the term "machine" should also be construed to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0084] As described herein, examples may include or operate on a processor, logic or multiple components, modules or mechanisms ("modules" herein). A module is a tangible entity (e.g., hardware) that can perform a specified operation and can be configured or arranged in some way. In an example, a circuit can be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuits) as a module. In an example, one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors in their entirety or in part can be configured as a module by firmware or software (e.g., instructions, application parts or applications), and the module operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. The software causes the hardware to perform the specified operation when executed by the underlying hardware of the module.

[0085] Thus, the term "module" is understood to encompass at least one of a tangible hardware or software entity, which refers to an entity that is physically constructed, specially configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a specified manner or to perform part or all of any operation described herein. Considering examples where modules are temporarily configured, each of the modules does not need to be instantiated at any one time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time instance and to constitute different modules at different time instances.

[0086] The machine (e.g., computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1004, and a static memory 1006, some or all of which may communicate with each other via an interconnect (e.g., a bus) 1008. The machine 1000 may further include a display unit 1010 (shown as a video display), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display unit 1010, the input device 1012, and the UI navigation device 1014 may be a touch screen display. The machine 1000 may additionally include a mass storage device (e.g., a drive unit) 1016, a signal generating device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1022. Examples of sensors 1022 include one or more of a global positioning system (GPS) sensor, a compass, an accelerometer, temperature, light, camera, video camera, a sensor of physical state or position, a pressure sensor, a fingerprint sensor, a retinal scanner, or other sensors. The machine 1000 may also include an output controller 1024, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., printer, card reader, etc.).

[0087] The mass storage device 1016 may include a machine-readable medium 1026 on which is stored one or more sets of data structures or instructions 1028 (e.g., software) embodying or utilized by any one or more of the techniques or functionality described herein. The instructions 1028 may also reside, completely or at least partially, within the main memory 1004, within the static memory 1006, or within the hardware processor 1002 during execution of the instructions 1028 by the machine 1000. In an example, one or any combination of the hardware processor 1002, the main memory 1004, the static memory 1006, or the mass storage device 1016 may constitute a machine-readable medium.

[0088] Although the machine-readable medium 1026 is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database or associated cache or at least one of the servers) configured to store one or more instructions 1028. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 1000 and which causes the machine 1000 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory, and optical and magnetic media. Specific examples of machine-readable media may include nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and DVD-ROM disks. Machine-readable media include non-transitory machine-readable media. In some examples, machine-readable media may include machine-readable media that is not a transitory propagating signal.

[0089] The instructions 1028 may also be transmitted or received over a communication network 1032 using a transmission medium via the network interface device 1020. The machine 1000 may communicate with one or more other machines using any of a number of transmission protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a Plain Old Telephone (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as ), IEEE 802.15.4 series standards, Long Term Evolution (LTE) series standards, Universal Mobile Telecommunications System (UMTS) series standards, peer-to-peer (P2P) networks, etc. In an example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial or telephone jacks) or one or more antennas 1030 to connect to the communication network 1032. In an example, the network interface device 1020 may include multiple antennas 1030 to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO) or multiple-input single-output (MISO) technology for wireless communication. In some examples, the network interface device 1020 may use multi-user MIMO technology for wireless communication.

[0090] The features and flow charts described herein may be embodied in one or more methods as method steps or in one or more applications as described above. According to some configurations, an "application" or "applications" is a program that performs the functions defined in a program. Various programming languages ​​may be used to generate one or more applications that are structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system of a mobile phone or other mobile operating system. In this example, third-party applications can call API calls provided by the operating system to facilitate the functionality described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in specialized computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0091] The program aspects of the technology may be considered to be a "product" or "article of manufacture" typically in the form of at least one of executable code or associated data (these executable code or associated data are carried on or embodied in a type of machine-readable medium). For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or part of the software may sometimes be communicated over the Internet or various other telecommunications networks. For example, such communications may enable the software to be loaded from one computer or processor to another computer or processor. Therefore, another type of medium that can carry programming, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline telephone networks, and through various air links. Physical elements that carry such waves (such as wired or wireless links, optical links, or the like) may also be considered as media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0092] Thus, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as can be used to implement the AR device 110, back-end server 150, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including wires that make up the bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media thus include, for example: a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, a punched card tape, any other physical storage medium with a pattern of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read at least one programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0093] The scope of protection is limited only by the claims that now follow. The scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims, which should encompass all structural and functional equivalents when interpreted in light of this specification and subsequent prosecution history. Nonetheless, none of the claims is intended to include subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0094] Except as stated immediately above, nothing stated or illustrated is intended or should be construed as causing a dedication to the public of any element, step, feature, object, benefit, advantage, or equivalent, regardless of whether or not it is recited in the claims.

[0095] It should be understood that the terms and expressions used in this article have the ordinary meanings assigned to the corresponding fields of inquiry and research corresponding to such terms and expressions, unless the specific meanings are otherwise set forth herein. Relational terms such as first and second can be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprise", "comprising", "include", "including" or any other variations thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a list of elements or steps includes not only those elements or steps, but also includes other elements or steps inherent to such processes, methods, articles or devices that are not explicitly listed. In the absence of further restrictions, in the process, method, article or device including the element, an element starting with "one" or "an" does not exclude the presence of additional identical elements.

[0096] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions and other specifications set forth in this specification (including in the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and the customary practices in the art to which they pertain. For example, parameter values ​​or the like may vary from the stated quantities by as much as ±10% unless expressly stated otherwise.

[0097] In addition, it can be seen in the foregoing detailed description that various features are grouped together in various examples for the purpose of simplifying the present disclosure. The disclosure method should not be interpreted as reflecting the intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, and each claim exists independently as a separately claimed subject matter.

[0098] While the foregoing has described what is considered to be the best mode and other examples, it should be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples and may be applied to many applications, only some of which have been described herein. It is intended that any and all modifications and variations that fall within the true scope of the present concept be protected by the following claims.

Claims

1. A method for interacting an augmented reality (AR) device with an Internet of Things (IoT) device, comprising: Receiving IoT device information in a standardized format, the IoT device information representing at least one device input and available output information for the IoT device; determining, from the received IoT device information, a predetermined AR user interface widget to be presented for the IoT device, the predetermined AR user interface widget converting a user input to the AR device into the at least one device input for the IoT device; Receiving a selection of the IoT device; providing the predetermined AR user interface widget for the selected IoT device to the AR device as an overlay on a display of the AR device; receiving the at least one device input for the IoT device in response to a user interaction with the predetermined AR user interface widget displayed by the AR device; as well as The at least one device input is provided to the IoT device. 2 . The method of claim 1 , further comprising receiving location information from the AR device.

3. The method of claim 2, further comprising selecting the IoT device based on a proximity of the AR device to the IoT device. 4 . The method of claim 2 , further comprising selecting the IoT device based on verbal input from the user or based on an orientation of the AR device toward the IoT device. 5 . The method according to claim 1 , further comprising automatically presenting the predetermined AR user interface widget after selecting the IoT device. 6 . The method of claim 1 , further comprising receiving status data from the IoT device and providing the status data to a display of the AR device.

7. The method according to claim 1, wherein: Receiving AR device input in response to user interaction with the predetermined AR user interface widget displayed by the AR device includes receiving a gesture or body posture from the user.

8. The method according to claim 1, wherein: Receiving IoT device information includes registering the IoT device in response to the IoT device broadcasting its presence to the backend server after power-on or the backend server periodically polling for new IoT devices.

9. The method according to claim 1, wherein: Providing the predetermined AR user interface widget for the selected IoT device as an overlay on a display of the AR device includes: upon selecting the IoT device, the IoT device launches a preferred AR overlay that communicates with the AR device.

10. The method according to claim 1, wherein: Determining the predetermined AR user interface widget to be presented for the IoT device includes: comparing the received IoT device information with available standardized AR user interface widgets to determine at least one standardized AR user interface widget that can interpret the IoT device input and output, and selecting a standardized AR user interface widget from the available standardized AR user interface widgets that converts user input to a user input device into at least one of the following forms of IoT device input according to the form of the IoT device input of the IoT device: Boolean, range value, two-dimensional relative value, two-dimensional absolute value, two-dimensional path, three-dimensional path, red, green, blue (RGB) color input, time or text.

11. An augmented reality (AR) device that interacts with an Internet of Things (IoT) device, the AR device comprising: monitor; User input devices; a registration module that receives IoT device information in a standardized format, the IoT device information representing at least one device input and available output information for the IoT device; an AR widget user interface, the AR widget user interface determining a predetermined AR user interface widget to be presented for the IoT device from the received IoT device information, the predetermined AR user interface widget being provided to the AR device as an overlay on the display, the predetermined AR user interface widget being adapted to convert a user input from the user input device into the at least one device input for the IoT device; as well as A communication module is provided to the IoT device to provide the at least one device input converted from the user input to the user input device.

12. The AR device according to claim 11, further comprising a processor implementing an operating system, wherein: The registration module and the communication module are implemented by the processor via an application programming interface or a system layer of the operating system.

13. The AR device according to claim 11, wherein: The AR widget user interface maps multiple AR devices to corresponding AR widget user interfaces for the IoT device.

14. The AR device according to claim 13, wherein: The AR widget user interface draws graphics and specifies gesture data transformations required to control the corresponding AR widget user interface for the IoT device.

15. The AR device according to claim 11, wherein: The communication module receives the location information of the AR device and selects the IoT device based on the proximity of the AR device to the IoT device.

16. The AR device according to claim 11, wherein: The communication module receives verbal input from the user or orientation data regarding an orientation of the AR device toward the IoT device, and selects the IoT device based on the verbal input from the user or based on the orientation data.

17. The AR device according to claim 11, wherein: When the AR device is within a predetermined distance from the IoT device, the AR widget user interface automatically presents the predetermined AR user interface widget to the AR device.

18. The AR device according to claim 11, wherein: The communication module receives status data from the IoT device and provides the status data to a display of the AR device.

19. The AR device according to claim 11, wherein: The AR widget user interface selects the AR user interface widget to be presented for the IoT device by comparing the received IoT device information with the available standardized AR user interface widgets to determine at least one standardized AR user interface widget that can interpret the IoT device input and output, and selects a standardized AR user interface widget from the available standardized AR user interface widgets that converts the user input to the user input device into at least one of the following forms of IoT device input according to the form of the IoT device input of the IoT device: Boolean, range value, two-dimensional relative value, two-dimensional absolute value, two-dimensional path, three-dimensional path, red, green, blue (RGB) color input, time or text.

20. A non-transitory computer-readable medium having instructions stored thereon, wherein: Execution of the instructions by one or more processors causes the one or more processors to interact an augmented reality (AR) device with an Internet of Things (IoT) device by performing operations including: Receiving IoT device information in a standardized format, the IoT device information representing at least one device input and available output information for the IoT device; determining, from the received IoT device information, a predetermined AR user interface widget to be presented for the IoT device, the predetermined AR user interface widget converting a user input to the AR device into the at least one device input for the IoT device; Receiving a selection of the IoT device; providing the predetermined AR user interface widget for the selected IoT device to the AR device as an overlay on a display of the AR device; receiving the at least one device input for the IoT device in response to a user interaction with the predetermined AR user interface widget displayed by the AR device; as well as The at least one device input is provided to the IoT device.

Images