AR system for providing interactive experience in smart space

Abstract

The interactive experience is generated using an augmented reality (AR) system that utilizes native input and output functionality of the AR device to generate an immersive and interactive experience in a smart space that combines programming logic of a visual representation, AR content, and IoT devices. The system connects IoT devices to each other in a native AR environment, provides virtual content and logic gates, script sequences, and action cycles, solves the visibility problem of IoT devices that are far away from each other, and designs logic gates and flow circuits that best support user intentions when generating interactive experiences in smart space. The interactive experience is generated by using a program in a visual programming interface of the AR device that generates a behavior of the selected IoT using logic gates and control circuitry connected to the IoT device. Once completed, the program of the behavior is executed by the runtime module to control the IoT device during the interactive experience.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 841,397, filed on June 15, 2022, and is related to U.S. application serial number 17 / 841,365, filed on June 15, 2022, entitled “STANDARDIZED AR INTERFACES FOR IoT DEVICES,” the contents of both of which are fully incorporated herein by reference. Technical Field

[0003] The present subject matter relates to systems and methods for enabling users of augmented reality (AR) systems to interact with Internet of Things (IoT) devices, and more particularly, to systems and methods for providing an immersive and interactive experience combining programming logic of visual representations, AR content, and IoT technology. Background Art

[0004] The so-called "Internet of Things" or "IoT" is a network of physical objects embedded with sensors, software, and other technologies for enabling connectivity and data exchange with other devices via the Internet. For example, IoT devices are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided that act as smart home hubs for connecting different smart home products. IoT devices have also been used for many other applications. Application layer protocols and supporting frameworks for implementing such IoT applications have been provided. Artificial intelligence is also combined with IoT infrastructure to enable more efficient IoT operations, improve human-machine interactions, and enhance data management and analysis.

[0005] An increasing number of IoT devices have become available to home consumers as well as public and private entities. These IoT devices range from smart red, green, and blue (RGB) light bulbs to automatic plant watering systems. At the same time, the adoption of AR technology is growing, for example in Currently, augmented reality (AR) is mainly used to enhance photos and videos, and is also used in interactive applications (e.g. However, there is little work on the topic of developing tools for producers of interactive experiences that leverage both AR and IoT technologies. Summary of the invention BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The features of the various embodiments disclosed will be readily understood from the detailed description made below with reference to the attached drawings. Reference numerals are used with each element in the specification and throughout the multiple views of the drawings. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements, with the additional lower case letters referring to the specific element.

[0007] The various elements shown in the drawings are not drawn to scale unless otherwise specified. The sizes of the various elements may be enlarged or reduced for clarity. The various figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following figures:

[0008] Figure 1 is a general block diagram of a system for connecting an augmented reality (AR) camera device to an Internet of Things (IoT) device;

[0009] Figure 2 In the sample configuration Figure 1 A block diagram of the system's components and information flow;

[0010] Figure 3 is a sample configuration showing Figure 2 A block diagram of components of an experience generation user interface;

[0011] Figure 4A is a diagram of attaching a control widget to an IoT device in a sample configuration to generate a two-dimensional (2D) representation of a complex custom component;

[0012] Figure 4B is a diagram of attaching a control widget directly to an IoT device in a sample configuration to generate a 2D representation of a complex custom component;

[0013] Figure 5 is a diagram depicting sampling the state of an IoT device and pasting the sampled state into a sequencer component in a sample configuration;

[0014] Figure 6 is a diagram depicting a 2D example of a gateway node that forwards connections between spatially separated components in a sample configuration;

[0015] Figure 7 is a diagram depicting a 2D example of an IoT device proxy in a sample configuration that forwards connections between an actual device (left) and a proxy representation of the actual device at a spatially separated location;

[0016] Figure 8 is a flow chart of a method for producing an immersive and interactive experience using an augmented reality (AR) device in a smart space having multiple IoT devices in a sample configuration;

[0017] Fig. 9 is a block diagram showing a sample configuration of a mobile phone suitable for operating as an AR camera device in the sample configuration; and

[0018] Fig.10 is a block diagram illustrating a sample configuration of a computer system suitable for implementing the back-end server systems and methods described herein. DETAILED DESCRIPTION

[0019] Augmented reality (AR) devices and Internet of Things (IoT) devices are now available in many homes and public and private entities. This makes it possible to create interactive "smart" spaces that combine both AR and IoT technologies. However, there is little work to develop tools for creators of interactive experiences that wish to utilize AR and IoT technologies at the same time. The following disclosure describes a system that utilizes the native input and output capabilities of AR devices to create an immersive and interactive experience in a smart space that combines programming logic for visual representations, AR content, and IoT devices. The following disclosure also describes user interfaces and interaction techniques used by the system to implement multiple common programming operations supported by the system.

[0020] To build an AR interface for generating interactive experiences in a "smart" space, the systems and methods described herein connect IoT devices to each other in a native AR environment, provide virtual content and logic gates, script sequences and action loops, address visibility issues for IoT devices that are far apart, and design logic gates and flow circuits that best support user intent when generating interactive experiences in smart spaces. The interactive experience is generated by generating a program of the behavior of the selected IoT using logic gates and control circuits connected to the IoT devices in a visual programming interface of the AR device. Once completed, the program of the behavior is executed by a runtime module to control the IoT device during the interactive experience.

[0021] The methods described herein enable a user to interact with at least one Internet of Things (IoT) device using an augmented reality (AR) device to generate an interactive experience. During the interactive experience, the user provides a selection of at least one IoT device to be controlled by the AR device, and an AR user interface widget or graphical representation for the selected at least one IoT device is provided to the AR device as an overlay on the display of the AR device. As used herein, a "widget" is a component of an application or interface that enables a user to perform a function or access a service. The interactive experience is programmed by using a program in a visual programming interface of the AR device that generates the behavior of the selected at least one IoT device using at least one of the logic gates or control circuits connected to the selected at least one IoT device. The logic gates and control elements together specify the behavior to be performed by the IoT device in the visual programming interface.

[0022] The program of the behavior of the selected at least one IoT device is based on input / output rules for the at least one IoT device, where the input / output rules can be expressed via Boolean logic, arithmetic functions, n-ary mappings, or mathematically definable operations. Programming can also include selecting control circuits to express functions including at least one of thresholds, clamping functions, or oscillators, and generating a sequence of actions performed by the at least one IoT device during the interactive experience. Connectors can also be accessed in a visual programming interface of the AR device to connect corresponding IoT devices, logic gates, and control circuits to specify program logic and information flow used in the interactive experience. The program of the behavior is executed during runtime to selectively control at least one IoT device during the interactive experience. The program of the behavior of the interactive experience can be debugged and replayed via a debugging and playback user interface of the AR device.

[0023] The AR device for generating an interactive experience described herein may include a runtime module that determines the type of AR user interface widget or graphical representation to be presented for at least one IoT device on a display of the AR device, the runtime module receiving a selection of at least one IoT device to be controlled by the AR device during the interactive experience, and providing the determined AR user interface widget or graphical representation for the selected at least one IoT device as an overlay to the display. The experience generation user interface provides a visual programming interface through which a user programs the interactive experience by using at least one of the logic gates or control circuits connected to the selected at least one IoT device in the visual programming interface to generate a program for the behavior of the selected at least one IoT device. The logic gates and control circuits together specify the behavior to be performed by the IoT device. Upon completion, the runtime module executes the program of behavior in the runtime module to selectively control at least one IoT device during the interactive experience programmed by the user.

[0024] In a sample configuration, the experience generation user interface is a builder component that includes a user interface component that includes at least one of: (a) an IoT device user interface for interacting with at least one IoT device to attach logic gates or control circuits to inputs or outputs of at least one IoT device; (b) a logic gate / control circuit user interface that provides access to logic gates for programming behavior based on input / output rules of the IoT device that can be expressed via Boolean logic, arithmetic functions, n-ary mappings, or mathematically definable operations, and access to control circuits to express functions including at least one of thresholds, clamping functions, or oscillators; or (c) a sequencer user interface for defining a sequence of actions performed by at least one IoT device as part of an interactive experience.

[0025] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products, which illustrate the examples set forth in the present disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without such details. The aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in a variety of ways. The terms and nomenclature used herein are only used to describe the purpose of specific aspects and are not intended to be restrictive. In general, known instruction instances, protocols, structures, and techniques do not need to be shown in detail.

[0026] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links or the like, by which an electrical or magnetic signal generated or supplied by one system element is communicated to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communications media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0027] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art after reviewing the following and the accompanying drawings, or may be learned by production and operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, means and combinations particularly pointed out in the appended claims.

[0028] The convergence of connected IoT and AR devices in home and commercial spaces offers tremendous opportunities to generate novel experiences in environments referred to in this article as “smart” spaces. Smart spaces combine real-world and virtual elements by fusing artificially displayed AR content with IoT device outputs and through interactions with IoT devices connected to the real world. However, coordinating such a fusion of AR devices, virtual content, and the physical outputs (visual, auditory, tactile) of IoT devices is a technical challenge.

[0029] Known systems do not involve the use of AR and spatial user interfaces to facilitate the generation of interactive experiences in smart spaces based on native AR user interfaces as described herein. Examples of AR interactive experiences described herein range from a game of "hide and seek" for children in a family, an interactive museum exhibit, or a large and complex scene such as a theme park ride or an AR escape room with IoT device controls where multiple users can share a space and manipulate IoT devices to make desired adjustments to the environment.

[0030] The systems and methods described herein include a set of user interfaces that enable such systems and methods to function effectively for their users. A native AR system is used because it provides many advantages. For example, since no additional abstraction layer is required, such as a desktop-based editor or integrated development environment, the native AR system enables users to immediately see the results of their work. In addition, the spatial layout of all components as well as their connections and any logic are immediately obvious when they are co-located in the smart space, so it is not necessary to explain the visualization of the smart space. Since IoT devices can interact directly, the experience can be adjusted in real time to view the experience in the right location. The native AR system also allows multiple users to collaboratively generate smart spaces in the same environment.

[0031] AR devices are spatially aware, and IoT devices are often distributed in the user's space. These characteristics make it possible to select and manipulate a specific IoT device in AR by pointing to its location in space, rather than selecting it from a (potentially long) list of available devices in two dimensions. IoT devices may have an expanded set of inputs or multi-dimensional inputs. Spatial interactions using AR have the potential to extend beyond the screen of a mobile device while still being understandable and navigable by the user. In addition, spatial input modalities such as gestures are useful for inputting multi-dimensional variables (e.g., RGB color selection) compared to the options provided by a 2D touchscreen.

[0032] To exploit these features, a tool is provided that enables AR interactive experiences with IoT devices to be generated for users of AR devices in a native AR environment by using input technologies native to AR (e.g., gestures). The architectural design and specifications of the necessary components and their interoperation as described below can be used to generate interactive experiences in smart spaces. The architectural design can also include direct logic gate attachment, gateway nodes, and device agents, which facilitate the development of interactive experiences in large three-dimensional (3D) AR environments without cluttering the 3D AR environment.

[0033] Now refer to the attached Figures 1 to 10 An example is illustrated in and discussed below.

[0034] The system includes one or more IoT devices, one or more AR devices, and a backend server, which includes a registration module and a runtime module, which includes a program module. The backend server also performs state updates and logic evaluations and dispatches commands. It is understood that the backend server can run on a separate machine or as an application programming interface (API) or system layer on the operating system of any one of the one or more AR devices. In a sample configuration, the system also includes an experience generation user interface (UI) on the AR device, which interfaces the AR device with the IoT device UI, and the IoT device UI in turn interacts with the direct control widgets, input and output ports of the IoT device and is directly attached to the logic and program flow circuits. In other configurations, the system also includes a logic gate UI, a sequencer UI, a connector mechanism and a visualization element, and an IoT device agent. The system may also include a debugging and playback UI, which provides the user with the ability to debug the generated experience and playback the generated experience.

[0035] Figure 1 1 is a general block diagram of a system 100 for interacting an augmented reality (AR) camera device with an Internet of Things (IoT) device in a sample configuration. As shown, a plurality of AR camera devices 1101 to 110 N (such as suitable for including messaging applications (such as those available from Snap Inc. of Santa Monica, California ) of the Snapchat camera) respectively include AR displays 1201 to 1202 for displaying AR objects N The AR camera device 110 may include a smartphone, a laptop computer, a desktop computer, an electronic eyewear device, a head up display, or any other computing device that enables communication via the Internet. N It may further be adapted to include software 1301 to 130 N To support features such as hand and body recognition and tracking, marker tracking in AR space, and the like. Such features are readily available in messaging applications, such as the aforementioned Messaging apps.

[0036] In some examples, corresponding AR camera devices 1101 to 110 NA common AR widget user interface 140 may be shared. The AR widget user interface 140 may implement a user interface widget for interacting with the IoT device 110 in a sample configuration. The AR widget UI 140 may be used as a widget API layered between the AR application and the operating system of the AR device 110. It is understood that in the sample configuration, the AR user interface widget and related object recognition services may also be implemented by each AR camera device 1101 to 110 N Software running on 1301 to 130 N To be performed, and in a sample configuration, a common AR widget user interface 140 may be provided separately for each AR camera device 110. More details about the AR widget user interface 140 may be found in the above-mentioned related applications.

[0037] If you will refer to Figure 2 As explained in more detail, the backend server 150 acts as a bridge between the AR camera device 110 and the physical IoT device 160. The backend server 150 can also send commands to the IoT device 160 to act in any number of ways supported by the corresponding IoT device 160. The IoT device 160 can be controlled to respond to the AR camera device 1101 to 110 N The user interacts with the AR camera device 1101 to 110 and performs the desired physical action 180 and provides the sensor output to initiate the communication with the AR camera device 1101 to 110 N One-way or two-way interaction. Such two-way interaction may include transmitting status information from IoT device 160 to the corresponding AR camera device 110. Backend server 150 may also respond to context information provided by other context servers 170. Such context information may include time of day, weather conditions, global positioning system (GPS) data, and any other information available from a third-party server. As noted above, backend server 150 may run on a separate machine or as a separate server for AR camera devices 1101 to 110. N An application programming interface (API) or system layer on an operating system of any of the.

[0038] Figure 2 In the sample configuration Figure 1 1 is a block diagram of the components and information flow of the system 100. As shown, the system 200 includes an IoT device 160, an AR device 110, a backend server 150, an experience generation user interface 210, and a debugging and playback user interface 220. As further described below, the system 200 enables users of AR devices to generate immersive and interactive experiences in a "smart" space.

[0039] Generating an immersive and interactive experience in a "smart" space involves programming specified behaviors of one or more IoT devices 160 in the space in a native AR programming environment. IoT devices 160 can be any available connected smart device known in the art. For example, IoT devices 160 can include common household devices or semi-professional devices that end users may encounter in their homes or commercial spaces, such as RGB smart lights, thermostats, smart speakers, smart switches, and the like. Those skilled in the art will appreciate that as long as such devices are connected via Or other communication channels that enable Internet access provide connections to the AR device 110 and the backend server 150, and the IoT device also provides active or passive positioning in the environment that can be received by the AR device 110, such IoT devices 160 can be suitable for working in the system described herein. Active positioning can be achieved through radiofrequency (RF) beacons, ultrasonic waves, visible light signals, or via a global positioning system (GPS) for outdoor applications. On the other hand, passive positioning can be achieved by saving the fixed position of the IoT device 160 in the memory of the backend server 150 or the AR device 110 or through a visual code near the IoT device 160 for selection or other trustees that the AR device 110 can recognize. The AR device 110 and the IoT device 160 can also perform limited duplex communication to convert user input into the AR device 110 into a display of an AR user interface widget suitable for the IoT device 160. The IoT device 160 can also provide information about their status to external devices via appropriate endpoints.

[0040] The AR device 110 is used to both generate the interactive experience and experience the generated interactive experience. Suitable AR devices 110 include, but are not limited to, smartphones, tablet computers, or wearable devices such as smart glasses. In a sample configuration, the AR device 110 is connected to the user's computer via a communication channel (e.g., The AR device 110 may provide a connection to the backend server 150 or directly to the IoT device 160 for sending and receiving messages, and also provide positioning capabilities from inside to outside or from outside to inside. In one example, the positioning capability of the AR device 110 generates a relative posture (position, orientation) between the AR device 110 and any IoT device 160 to be used in the interactive experience. In addition, in addition to standard input modes (e.g., touch screen, buttons, trackpad, voice, etc.), the AR device 110 can support input such as gesture or body posture tracking.

[0041] like Figure 2As shown, the backend server 150 includes two main modules, a registration module 152 and a runtime module 154. The registration module 152 registers the available command inputs and status outputs of the IoT devices 160 that are exposed for interaction with the AR device 110. The IoT devices 160 can broadcast their presence to the backend server 150 when powered on to initiate the registration process. During the registration process, the IoT devices 160 provide a list of their available command inputs and status inputs to the registration module 152. Conversely, the backend server 150 can periodically poll for new IoT devices 160. In addition, the inputs and outputs of the AR user interface widgets to be used for the corresponding IoT devices 160 can be specified during the registration process. The AR user interface widgets can be defined using a standardized schema, such as the following example JSON file, which shows a minimum configuration for controlling a smart light:

[0042] Similarly, the AR devices 110 may register with the backend server 150 using the registration module 152 and signal that they are ready to receive updates from the IoT device 160, and also register their ability to send commands directly to the backend server 150 or the IoT device 160. The AR devices 110 may send their current locations to the backend server 150 along with the updates, and the backend server 150 may coordinate certain updates between the IoT device 160 and the AR device 110 (e.g., start sending a particular state to the AR device 110 or activate a certain input AR widget user interface (UI) 140 on the AR device 110). However, some parts of such logic may also run locally on the AR device 110.

[0043] The runtime module 154 executes program logic to play back an interactive experience (IE) program generated by the user and representing an interactive experience. The runtime module 154 can implement the IE program using one or more IE program modules that control a portion of the user experience. The IE program module can maintain an internal state that is updated by a state update and logic evaluation component that divides the IE program execution into discrete time steps. In each time step, the state update and logic evaluation component analyzes each IE program module and updates its state based on external input and its previous state, and sets a new output state (if available). If the new output state is derived from a state update, and this requires commands to be sent to the IoT device 160, these commands will be queued and dispatched later via the command dispatch component. The command dispatch component dispatches commands to the IoT device 160 using information from the registration module 152 and updates the representation of the state of the IoT device 160 in the IE program module using state information. In some configurations, the IoT device 160 can launch a preferred AR overlay for interacting with the IoT device 160, which is communicated to the AR device 110 by the runtime module 154.

[0044] It will be appreciated that the backend server 150 may run on a separate machine, or as a further API or system layer on the AR device operating system of the AR device 110 itself.

[0045] In a sample configuration, an experience generation user interface (UI) 210 is used by a user to construct an interactive experience. Figure 3 As shown, the experience generation UI 210 is a builder component that includes a user interface component that includes at least an IoT device UI 300, a logic gate / control circuit UI 310, and a sequencer UI 320, as well as a connector mechanism 330 and an IoT device agent 340, which is a virtual representation of an IoT device 160 at a remote location. A user can select each of these builder components of the experience generation UI 210 and use the builder component as a tool in a visual programming interface to build corresponding features of an interactive experience to be implemented during runtime. Each of these components will be described in more detail below.

[0046] The IoT device UI 300 can be used by the user to interact directly with the IoT device 160 (e.g., in the case where the user wants to try direct setup) and also to attach input and output connections or logic gates. The IoT device UI 300 is a builder component that enables the user to implement multiple features, including direct control of widgets, input ports, output ports, and can also provide direct attachment of logic gates and IE program flow circuits. In a sample configuration, the IoT device UI 300 is shown on the user's AR device 110 when the user moves into the vicinity (or "interaction zone") of the IoT device 160, or when the user works with the IoT device agent 340 of the IoT device 160 at a remote location.

[0047] In a sample configuration, a direct control widget can be used to directly set any property exposed by the IoT device 160 using an input / output connector for a corresponding logical device. The direct control widget can be a customized per-device UI or generically generated by registering the input variable types of the IoT device 160 with the registration module 152, as described in the above-mentioned related applications. Such connectors and devices can be presented to the user's AR display through the operation of a spatially located visual programming tool (such as Ivy), which uses immersive virtual reality to generate logical program structures, establish logical links between smart objects, and visualize real-time data flows between real-world sensors and actuators. Spatially located visual programming tools (such as Ivy) use 2D visual programming environments such as CHIMP, CaveCAD, Scratch, Alice, WoTKit, Wyliodrin, and Node-RED to provide the required graphical tools to help users understand the logical connections between objects. Visual representations can also be obtained in many professional tools (such as LabView, SimuLink, Max, and Grasshopper for Rhino). The reality editor can be used for spatial context visual programming, which enables basic connections between smart objects created using AR on handheld devices. It will also be understood that the visual and spatial programming system described by Cao et al. can be adapted as a visual programming interface for programming IoT devices 160 as described herein.

[0048] To receive control commands from other components, the graphical AR representation of the IoT device 160 may have input ports that can be connected to other IoT devices 160 or logic gates. At each state update of the runtime module 154, all input values ​​may be updated according to the defined program flow or program logic.

[0049] Some IoT devices 160 may expose state (e.g., readings from smart air quality sensors, smart thermometers, and the like) that may serve as input for other parts of the experience. Thus, there may be input ports that may be connected to other IoT devices 160 or output ports of logic gates or program flow circuits. At each state update of a runtime module 154, all output values ​​may be updated according to the change in state of a particular IoT device 160.

[0050] To reduce clutter in the display of the AR device 110, which is particularly important when working in a 3D environment, certain logic gates and program flow circuits can be attached directly to the input ports of the IoT device 160 via their output ports, in a sense "piggy-backing" on them. Such "piggy-backing" reduces the requirement for the user to generate additional connections between the logic gates and program flow circuits and the inputs of the IoT device 160. Similarly, the inputs of the logic gates and program flow circuits can be attached directly to the outputs of the IoT device 160, with the same goal of reducing the number of connections to be drawn by the user. Direct attachments can also be stacked to build complex custom parts.

[0051] For example, Figure 4A 4 is a diagram of a two-dimensional (2D) representation 400 of a graphical representation 420 of an IoT device 160 that attaches a control widget 410 to a connection line 425. A plurality of such control widgets 410 indicating logic and control circuits can be connected to a graphical representation 420 of an IoT device 160 to assemble a logic circuit that specifies the operation of the IoT device 160. The resulting logic circuit can be simple or complex. The control widgets 410 can be combined in this way to generate a custom component in a sample configuration. In this example, the control widget 410 is a logic gate in the form of a conventional AND gate and the IoT device 160 is assumed to be a light switch. The graphical representation 420 virtually represents a light switch in an AR space. In this configuration, the light switch (IoT device 160) is turned on (ON) or off (OFF) in response to the output of the AND gate control widget 410, which provides an ON or OFF signal in response to its corresponding input.

[0052] Figure 4B4 is a diagram of a 2D representation 430 of a control widget 410 attached directly to a graphical representation 420 of an IoT device 160 without connection lines 425. In this example, the control widget 410 is again a logic gate in the form of a conventional AND gate. However, in this example, the control widget 410 is attached directly to an input port of the graphical representation 420 of the IoT device 160 without connection lines 425. This configuration enables a user to generate stackable, complex, customized components without drawing additional connections (e.g., by eliminating connection lines 425 between components), thereby reducing clutter on the display of the AR device 110 that might otherwise result from the generation of such components.

[0053] Understandably, Figure 4A and Figure 4B The custom parts shown in FIG. 1 can be displayed as AR objects on the display of the AR device 110 , as opposed to a desktop computer, and can be generated in the 3D AR domain as the user moves throughout the real-world domain.

[0054] Return to reference Figure 3 , the logic gate / control circuit UI 310 provides access to logic gates that enable users to program behavior based on input / output rules of the IoT device 160, which can be represented via Boolean logic, arithmetic functions, or any other mathematically definable n-ary mapping or operation. Control circuits can be used to express more complex functions, such as thresholds, clamping functions, or even oscillators. Designing such elements for AR interfaces requires careful consideration to make them as useful as possible. In their simplest form, logic gates and simple functions can be presented as 3D boxes with suitable attachment points for input and output connections. Symbols in front of the 3D boxes can be used to describe their behavior (e.g., AND). However, more complex control circuits (e.g., thresholds) can have their own user interfaces to enable users to enter desired control values ​​within a pre-established range of values. This can be achieved, for example, by using standardized AR user interface widgets of the type described in the aforementioned related applications, such as sliders or dials for entering values ​​and displays showing the selected values.

[0055] Depending on the input / output capabilities of the AR device 110 , the user's interaction with the control circuitry can be very realistic, for example, employing force feedback using a haptic device to give a visceral feeling of sliding a slider or turning a dial.

[0056] In some cases, a user may define a sequence of actions to be performed by the IoT device 160 as part of an interactive experience, such as flashing a light in a specified color sequence and playing a sound at the end. To store such sequences (essentially sequences of device states), a dedicated programming circuit called a sequencer is provided that is capable of storing a sequence of states from one or more IoT devices 160 in a timeline. The sequencer UI 320 provides user access to the sequencer programming circuitry.

[0057] Entering and representing state within a sequencer can be a challenge, especially in an AR context. Figure 5 As shown, one possible solution to this challenge is to expose a "sample" tool for the sequencer UI 320 of the sequencer component, through which the user can sample the state of the IoT device 160 by moving near the IoT device 160 and then applying the tool. The tool then brings the state to a representation on the state sequencer 530 (at least a text value, e.g., RGB (255, 0, 255) for a smart light bulb). The user can then move back and forth or copy and reuse this state on the sequencer timeline when building a sequence for their project.

[0058] For example, Figure 5 5 is a diagram depicting sampling the state of the IoT device 160 and pasting the sampled state into the sequencer component 530. Figure 5 In the example of , a state sequencer 530 can provide RGB fixture colors 500 that a user can sample by touching the color with a virtual sample selection device 510 (such as a stick). The user can paste the selected color into the state sequence 520 of the state sequencer 530 by touching the virtual sample selection device 510 to the next state 540 in the sequence 520. The output of the sequencer 530 in this example can be provided to the RGB fixtures to sequence through the fixture colors in the specified sequence. In this way, the user can use the displayed AR object to control the order of the colors and the time when the colors are displayed by the IoT-enabled RGB fixture.

[0059] The connector mechanism 330 of the experience generation UI 210 includes building blocks of interactive experiences (IoT devices, logic gates, sequencers, etc.), with input and output connectors for connecting to the building blocks to specify program logic and information flow. Standard connectors used in existing visual programming environments use "wires" or "pipes" to connect components, which is also easy to implement in AR. However, in addition to "wires" or "pipes" for connecting components, some examples may also include gateway nodes ( Figure 6 ) and connectors that model their physical counterparts in AR.

[0060] In a sample configuration, a gateway node can be used to extend a smart space across a large area or volume to connect distant components to each other while minimizing visual clutter on the AR display 120 of the AR device 110. A gateway node is configured to logically connect two distant locations without the need to drag a line across the distance. A gateway node can have one or more connections and ensure that the connections are appropriately labeled to facilitate users extending the connection at the remote end.

[0061] Figure 6 is a diagram depicting a 2D example of a gateway node 600 that forwards connections between spatially separated Boolean components in a sample configuration. As shown, the gateway node 600 is a physically separate element that collects inputs in order to avoid extending the connection to the entire display of the AR device 110. In this example, the gateway node 600 is labeled OR #25 and IoT device #2 because these are the elements 620 and 630 for which the gateway node 600 extends connection 610 to connect to the AND gate 640. As shown, there may be multiple elements to the left of the OR gate 620, such as IoT device #1 (element 650). As an example, Figure 6 The configuration may be used to pipe an audio stream from an IoT device 160 in one room to an IoT device 160 in another room that is not within the field of view of the AR device 110 .

[0062] Assuming the availability of very high fidelity AR devices in terms of graphics, audio, and tactile feedback, connectors can be modeled very realistically to correspond to their physical counterparts. For example, connectors can represent the visual shape, sound, and feel of a connector to provide a realistic "snap" as a connection is established during the creation of an interactive experience. For example, a connector can model the plugging behavior of a 3.5mm audio jack or an RJ45 connector. In addition, based on the type of data being connected, specified audiovisual or tactile characteristics can be provided. In addition, virtual "resistance" may be added if the user attempts an invalid connection.

[0063] As noted above with respect to gateway node 600, it may be inconvenient to generate a connection to an IoT device 160 that is a significant distance from the component with which the user desires to establish a connection. Figure 7700 is a diagram depicting a 2D example of an IoT device proxy 710 selected from the IoT device proxies 340 that forwards a connection between an actual device 720 (left) and an IoT device proxy 710 representing the actual device 720 at a spatially separated location. The IoT device proxy 710 provides a proxy for the actual device to expose inputs and outputs to enable a user to generate a virtual representation of a single or group of IoT devices 160 and copy and paste the IoT device proxy 710 of the actual device 720 at different locations, with the goal of simplifying the process of establishing a connection.

[0064] The debug and playback UI 220 for the generated experience enables the user to play, pause, or reset the execution of the generated experience once the user has completed the generation of the interactive experience. Debug information can be displayed next to the components to explain their state, or even modify the state at runtime. In addition, the debug mode enables the user to pause and step through the experience and visually inspect the state of various components and the data currently being transmitted / received via outputs / inputs, respectively. For example, the display of the AR device 110 can display Boolean values ​​and the like during playback of the generated experience.

[0065] Figure 8 8 is a flow chart of a method 800 for generating an immersive and interactive experience using an AR device 110 in a smart space having multiple IoT devices 160 in a sample configuration. The method 800 may be implemented solely by the server 150 ( Figure 2 ), implemented by the AR device 110 implementing the server function, or implemented by a combination of the server 150 and one or more AR devices 110.

[0066] like Figure 8 As shown, method 800 includes, at 810, each IoT device 160 registering with server 150 by providing its device input and available output information. For example, the registration information can be provided using a standardized schema as described in the aforementioned related applications. At 820, server 150 determines the type of AR widget or other graphical representation to be presented for each IoT device 160 from the registration data provided by the IoT device 160. This information can be stored in a table on server 150 or in AR device 110. At 830, AR device 110 is also registered. At 830, location information for AR device 110 is also optionally obtained.

[0067] To program the interactive experience, at 840, the user provides the server 150 with a selection of IoT devices 160 to be controlled by the AR device 110 during the interactive experience. The selection may be automatic (e.g., based on the proximity of the AR device 110 to the IoT device 160) or may be made by the user based on verbal input from the user, orientation of the AR device 110 toward the IoT device 160, etc. At 850, an AR widget or other graphical representation for the selected IoT device 160 is provided to the AR device 110 as an overlay on the display of the AR device 110. The AR widget or other graphical overlay may appear automatically or may be selected by the user. Once the AR widget or other graphical representation has been displayed to the user, the user may program the behavior for the selected IoT device 160 at 860.

[0068] For example, as described above, at 860 the user can use the logic gate / control circuit UI 310 to access a graphical logic gate that enables the user to program behavior based on input / output rules that can be represented via Boolean logic, arithmetic functions, or any other mathematically definable n-ary mapping or operation. Control circuits can be selected to express more complex functions, such as thresholds, clamping functions, or even oscillators. Logic gates and simple functions can be presented as 3D boxes with suitable attachment points for input and output connections to other components and IoT devices 160. Symbols in front of the 3D boxes can be used to describe their behavior (e.g., AND). Depending on the input / output capabilities of the AR device 110, the user's interaction with the control circuit can employ force feedback using a haptic device to give an instinctive feeling of sliding a slider or turning a dial. The user can also use the sequencer UI 320 to access the sequencer to define a sequence of actions to be performed by the IoT device 160 as part of the desired interactive experience. The user can also use the connector mechanism 330 of the experience generation UI 210 to connect the building blocks of the interactive experience (IoT devices, logic gates, sequencers, etc.) to specify program logic and information flow. As noted above, standard connectors can include "wires" or "pipes" to connect components. However, more sophisticated connectors such as gateway nodes 600 and connectors that model corresponding physical counterparts in AR can also be used to generate interactive experiences.

[0069] Once the desired interactive experience has been programmed at 860 through the visual programming interface and appropriate placement of logic components, the runtime module 154 executes the resulting program logic to play back the "program" generated by the user that represents the interactive experience at 870. The program modules of the runtime module 154 can maintain internal state, which is updated by a state update and logic evaluation component that divides program execution into discrete time steps. In each time step, the state update and logic evaluation component analyzes each program module and updates its state based on external inputs and its previous state and also sets a new output state (if available). If the new output state results from a state update, and this requires commands to be sent to the IoT device 160, these commands can be queued and dispatched later via the command dispatch component as described above.

[0070] As expected, at 880, the user may debug and playback the generated interactive experience via the debug and playback UI 220. Once debugging is complete, the interactive experience is ready for playback to control corresponding IoT devices 160 in the smart space based on the execution of the program generated by the user on the user's AR device 110 by the runtime module 154.

[0071] In one use case example, a user is an escape room enthusiast and is currently generating his own AR-enhanced escape room experience. This experience combines virtual objects displayed on the head-mounted AR device 110 with user interaction using real-world smart devices (e.g., actuators, lights, speakers, etc.) connected to the AR device 110 via the backend server 150. The user uses Figure 2 The user uses the AR system 200 to generate and run his escape room experience, which currently consists of two rooms connected by a short hallway. The current experience the user is working on is a science fiction scenario in which the user must escape the brig (room #1) of an abandoned starship and regain control of the ship on its bridge (room #2). In the experience, the brig and the bridge are connected via a "turbolift."

[0072] The user is currently processing a portion of the experience where the user has just escaped from the brig and is traveling to the bridge using a turbolift. The turbolift is implemented as a pair of powered sliding doors at either end of a short hallway separating Room #1 and Room #2. To simulate turbolift travel during the experience, the turbolift is equipped with an IoT device 160 that includes two powered sliding doors (brig / bridge), a speaker system, a controllable LED strip (4 LEDs) to the left of the sliding doors (to simulate passing through decks while traveling on the lift), a control display indicating the current deck, and a "viewport" over which the AR content of the ship's experience is overlaid.

[0073] The user is wearing smart glasses as the AR device 110 and is implementing the turbolift portion of the experience directly in the escape room space using the AR system 200. The turbolift portion of the experience may be implemented as follows.

[0074] Once the user has managed to solve all the puzzles in the brig and has managed to open the turbolift door, the "turbolift" voyage begins. Once the player enters the lift space and presses "bridge" on the turbolift's control panel, the "turbolift travel" program sequence begins. The user implements this with a simple "if" gate that checks for the successful completion of the brig and button presses, and then starts the sequencer module it is connected to. To do this, the user enters the "turbolift" area, instantiates the sequencer module, and drags connections to it from the brig door, bridge door, speaker system, LED strip, display, and AR viewport. The sequencer rows corresponding to the devices are automatically populated. The user can manually add columns corresponding to certain timestamps (e.g., 0 seconds, 5 seconds, 10 seconds, 15 seconds, 20 seconds, 25 seconds) and define actions for each device connected to the sequencer. Each device provides the sequencer with appropriate metadata for the type of action that the user can select in the sequencer column (e.g., the LED strip supports "off", "loop slow", and "loop fast" actions). The user sets up the sequencer, as shown in Table 1, to simulate an AR-enhanced "turbolift" flight.

[0075] Table 1

[0076]

[0077]

[0078] The sequencer thus implements the steps of closing the door (0 seconds), the elevator leaving the confinement area and accelerating (5 seconds), the elevator taxiing along the outside of the ship (10 seconds, 15 seconds), the elevator reaching the bridge (20 seconds), and opening the bridge door (25 seconds). The IoT-enabled door provides physical access, the IoT-enabled speakers provide elevator motion and ambient sounds, the IoT-enabled LEDs provide simulation of elevator motion and deck passing, the IoT-enabled AR viewport provides elevator travel animation and images of the exterior of the ship, and the IoT-enabled control panel (sci-FI-UI) all work together to give the user the illusion of traveling on a turbolift. Once the sequencer is built, the user connects the "done" output of the sequencer to the "start" input of the module that launches the bridge portion of the escape room experience.

[0079] Of course, many other IoT and AR-enabled interactive experiences can be built based on the logic and control protocols established by the user using the AR system 200 described herein, by combining the IoT device 160 and the AR device 110 with a sequencer that establishes the experience using the connector mechanism 330 and the IoT device agent 340.

[0080] System Configuration

[0081] Fig. 9 A sample configuration of a mobile device 900 suitable for interacting the AR camera device 110 with the IoT device 160 in a sample configuration is shown. In particular, Fig. 9 900 is a high-level functional block diagram of an example mobile device 900 that a user may use as an AR camera device 110 as described herein. The mobile device 900 may include a flash memory 905 that stores programming to be executed by the CPU 910 to perform all or a subset of the functions described herein. For example, the flash memory may store AR camera software 915 and optionally include AR processing software 920 for execution by the CPU 910 to enable a user of the mobile device 900 to display and interact with AR objects presented by the AR camera device 110 as described herein. The mobile device 900 may further include a camera 975 that includes one or more visible light cameras (a first visible light camera and a second visible light camera with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 905 may further include a plurality of images or videos generated via the camera 975.

[0082] The mobile device 900 may further include an image display 930, a mobile display driver 935 for controlling the image display 930, and a display controller 940. Fig. 9 In the example of , the image display 930 may include a user input layer 945 (e.g., a touch screen) layered on top of or otherwise integrated into the screen used by the image display 930. Examples of touch screen type mobile devices that may be used include, but are not limited to, smart phones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, Fig. 9 Thus, a block diagram is provided showing an example mobile device 900 having a user interface including a touch screen input layer 945 for receiving input (via touch, multi-touch or gestures, etc., by hand, stylus or other implement) and an image display 930 for displaying content.

[0083] like Fig. 9As shown, the mobile device 900 includes at least one digital transceiver (XCVR) 950, shown as a WWAN XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 900 may also include additional digital or analog transceivers, such as a short-range transceiver (XCVR) 955 for short-range network communication, such as via NFC, VLC, DECT, ZigBee, or For example, the short-range XCVR 955 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard protocols for communications implemented in wireless local area networks, such as the IEEE 802.11 standard. One of the standards.

[0084] In order to generate location coordinates for locating the mobile device 900, the mobile device 900 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 900 may utilize either or both of the short-range XCVR 955 and the WWAN XCVR 950 to generate location coordinates for positioning. For example, a cellular network, or based on The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the mobile device 900 via the XCVR 950, 955 through one or more network connections.

[0085] The transceivers 950, 955 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 950 include, but are not limited to, transceivers configured to operate according to code division multiple access (CDMA) and third generation partnership project (3GPP) network technologies, including, for example, but not limited to, 3GPP type 2 (or 3GPP2) and LTE, sometimes referred to as "4G". The transceiver may also be incorporated into broadband cellular network technology known as "5G". For example, transceivers 950, 955 provide two-way wireless communication of information including digitized audio signals, still images and video signals, web page information for display and input associated with the web page, and various types of mobile message communications to / from the mobile device 900.

[0086] The mobile device 900 may further include a microprocessor that functions as a central processing unit (CPU) 910. A processor is a circuit having elements that are constructed and arranged to perform one or more processing functions, typically various data processing functions. Although discrete logic components may be used, examples utilize components that form a programmable CPU. The microprocessor, for example, includes one or more integrated circuit (IC) chips that integrate electronic components to perform the functions of the CPU 910. The CPU 910, for example, may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuits may be used to form a CPU 910 or processor hardware in a smartphone, a laptop computer, and a tablet computer.

[0087] The CPU 910 acts as a programmable host controller for the mobile device 900 by configuring the mobile device 900 to perform various operations, for example, according to instructions or programming executable by the CPU 910. For example, such operations may include various general operations of the mobile device 900, as well as operations related to programming for a messaging application and an AR camera application on the mobile device 900. Although the processor may be configured using hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing programming.

[0088] The mobile device 900 also includes a memory or storage system for storing programming and data. Fig. 9 In the example shown in , the memory system may include flash memory 905, random-access memory (RAM) 960, and other memory components 965 as needed. RAM 960 may be used as short-term storage for instructions and data being processed by CPU 910, for example, as working data processing memory. Flash memory 905 generally provides longer-term storage.

[0089] Thus, in the example of mobile device 900, flash memory 905 may be used to store programming or instructions for execution by CPU 910. Depending on the type of device, mobile device 900 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPAD devices), Windows Mobile, Amazon Fire OS, RIM BlackBerry OS, or the like.

[0090] Finally, the mobile device 900 may include an audio transceiver 970 that may receive audio signals from the environment via a microphone (not shown) and provide audio output via a speaker (not shown). The audio signals may be coupled with the video signals and other messages by a messaging application or social media application implemented on the mobile device 900. In addition, the audio signals may include verbal commands from the user regarding the IoT device (e.g., "turn on the lights").

[0091] The technology described herein can also be used with one or more computer systems described herein or with one or more other systems. For example, the various processes described herein can be implemented with hardware or software, or a combination of the two. For example, at least one of the processors, memories, storage, output devices, input devices, or communication connections discussed below can each be at least a portion of one or more hardware components. A dedicated hardware logic component can be constructed to implement at least a portion of one or more of the technologies described herein. For example and without limitation, such hardware logic components can include field programmable gate arrays (FPGAs), program-specific integrated circuits (ASICs), program-specific standard products (ASSPs), systems-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc. The applications of the devices and systems that can include various aspects can widely include various electronic and computer systems. The technology can be implemented using two or more specific interconnected hardware modules or devices with related control and data signals, which can communicate between modules and through modules, or as part of a specific application integrated circuit. In addition, the techniques described herein can be implemented by a software program executable by a computer system. As an example, implementations can include distributed processing, component / object distributed processing, and parallel processing. Moreover, virtual computer system processing can be constructed to implement one or more techniques or functions, as described herein.

[0092] Fig.10 A sample configuration of a computer system suitable for implementing the systems and methods described herein is shown. In particular, Fig.10A block diagram of an example of a machine 1000 is shown, on which one or more configurations of an AR camera device 110, an AR widget user interface 140, a backend server 150, and a context server 170 can be implemented. In alternative configurations, the machine 1000 can operate as a standalone device or can be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 can operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, the machine 1000 can act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. In a sample configuration, the machine 1000 can be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web device, a server, a network router, a switch or a bridge, or any machine capable of executing instructions (continuously or otherwise) specifying the actions to be taken by the machine. For example, the machine 1000 can be used as a workstation, a front-end server, or a back-end server of a communication system. The machine 1000 can implement the methods described herein by running software for implementing features as described herein to enable the AR device 110 to control the IoT device 160. In addition, although only a single machine 1000 is illustrated, the term "machine" should also be construed to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0093] As described herein, examples may include or operate on a processor, logic or multiple components, modules or mechanisms ("modules" herein). A module is a tangible entity (e.g., hardware) that can perform a specified operation and can be configured or arranged in some way. In an example, a circuit can be arranged in a specified manner (e.g., internally or relative to an external entity such as other circuits) as a module. In an example, one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors in their entirety or in part can be configured as a module by firmware or software (e.g., instructions, application parts or applications), and the module operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. The software causes the hardware to perform the specified operation when executed by the underlying hardware of the module.

[0094] Thus, the term "module" is understood to encompass at least one of a tangible hardware or software entity, which refers to an entity that is physically constructed, specially configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a specified manner or to perform part or all of any operation described herein. Considering examples where modules are temporarily configured, each of the modules does not need to be instantiated at any one time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time instance and to constitute different modules at different time instances.

[0095] The machine (e.g., computer system) 1000 may include a hardware processor 1002 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1004, and a static memory 1006, some or all of which may communicate with each other via an interconnect (e.g., a bus) 1008. The machine 1000 may further include a display unit 1010 (shown as a video display), an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display unit 1010, the input device 1012, and the UI navigation device 1014 may be a touch screen display. The machine 1000 may additionally include a mass storage device (e.g., a drive unit) 1016, a signal generating device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1022. Examples of sensors 1022 include one or more of a global positioning system (GPS) sensor, a compass, an accelerometer, temperature, light, camera, video camera, a sensor of physical state or position, a pressure sensor, a fingerprint sensor, a retinal scanner, or other sensors. The machine 1000 may also include an output controller 1024, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., printer, card reader, etc.).

[0096] The mass storage device 1016 may include a machine-readable medium 1026 on which is stored one or more sets of data structures or instructions 1028 (e.g., software) embodying or utilized by any one or more of the techniques or functionality described herein. The instructions 1028 may also reside, completely or at least partially, within the main memory 1004, within the static memory 1006, or within the hardware processor 1002 during execution of the instructions 1028 by the machine 1000. In an example, one or any combination of the hardware processor 1002, the main memory 1004, the static memory 1006, or the mass storage device 1016 may constitute a machine-readable medium.

[0097] Although the machine-readable medium 1026 is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database or associated cache or at least one of the servers) configured to store one or more instructions 1028. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 1000 and which causes the machine 1000 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory, and optical and magnetic media. Specific examples of machine-readable media may include nonvolatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and DVD-ROM disks. Machine-readable media include non-transitory machine-readable media. In some examples, machine-readable media may include machine-readable media that is not a transitory propagating signal.

[0098] The instructions 1028 may also be transmitted or received over a communication network 1032 using a transmission medium via the network interface device 1020. The machine 1000 may communicate with one or more other machines using any of a number of transmission protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a Plain Old Telephone (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as ), IEEE 802.15.4 series standards, Long Term Evolution (LTE) series standards, Universal Mobile Telecommunications System (UMTS) series standards, peer-to-peer (P2P) networks, etc. In an example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial or telephone jacks) or one or more antennas 1030 to connect to the communication network 1032. In an example, the network interface device 1020 may include multiple antennas 1030 to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO) or multiple-input single-output (MISO) technology for wireless communication. In some examples, the network interface device 1020 may use multi-user MIMO technology for wireless communication.

[0099] The features and flow charts described herein may be embodied in one or more methods as method steps or in one or more applications as described above. According to some configurations, an "application" or "applications" is a program that performs the functions defined in a program. Various programming languages ​​may be used to generate one or more applications that are structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on platforms such as IOS TM ANDROID TM , Mobile software running on a mobile operating system of a mobile phone or other mobile operating system. In this example, third-party applications can call API calls provided by the operating system to facilitate the functionality described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in specialized computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0100] The program aspects of the technology may be considered to be a "product" or "article of manufacture" typically in the form of at least one of executable code or associated data (these executable code or associated data are carried on or embodied in a type of machine-readable medium). For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memory of a computer, processor, or the like, or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or part of the software may sometimes be communicated over the Internet or various other telecommunications networks. For example, such communications may enable the software to be loaded from one computer or processor to another computer or processor. Therefore, another type of medium that can carry programming, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline telephone networks, and through various air links. Physical elements that carry such waves (such as wired or wireless links, optical links, or the like) may also be considered as media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0101] Thus, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as can be used to implement the AR device 110, back-end server 150, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including wires that make up the bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer readable media thus include, for example: a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, a punched card tape, any other physical storage medium with a pattern of holes, a RAM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read at least one programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0102] The scope of protection is limited only by the claims that now follow. The scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims, which shall encompass all structural and functional equivalents when interpreted in light of this specification and the subsequent prosecution history. Nonetheless, none of the claims is intended to include subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0103] Except as stated immediately above, nothing stated or illustrated is intended or should be construed as conferring upon the public a dedication of any element, step, feature, object, benefit, advantage, or equivalent, whether or not recited in the claims.

[0104] It should be understood that the terms and expressions used in this article have the ordinary meanings assigned to the corresponding fields of inquiry and research corresponding to such terms and expressions, unless the specific meanings are otherwise set forth herein. Relational terms such as first and second can be used only to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprise", "comprising", "include", "including" or any other variations thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a list of elements or steps includes not only those elements or steps, but also includes other elements or steps inherent to such processes, methods, articles or devices that are not explicitly listed. In the absence of further restrictions, in the process, method, article or device including the element, an element starting with "one" or "an" does not exclude the presence of additional identical elements.

[0105] Unless otherwise indicated, any and all measurements, values, ratings, positions, amplitudes, dimensions and other specifications set forth in this specification (including in the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and the customary practices in the art to which they pertain. For example, parameter values ​​or the like may vary from the stated quantities by as much as ±10% unless expressly stated otherwise.

[0106] In addition, it can be seen in the foregoing detailed description that various features are grouped together in various examples for the purpose of simplifying the present disclosure. The disclosure method should not be interpreted as reflecting the intention that the claimed examples require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected does not lie in all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, and each claim exists independently as a separately claimed subject matter.

[0107] While the foregoing has described what is considered to be the best mode and other examples, it should be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples and may be applied to many applications, only some of which have been described herein. It is intended that any and all modifications and variations that fall within the true scope of the present concept be protected by the following claims.

Claims

1. A method for generating an interactive experience using an augmented reality (AR) device interacting with at least one Internet of Things (IoT) device, comprising: receiving a selection of at least one IoT device to be controlled by the AR device during the interactive experience; providing, to the AR device, an AR user interface widget or graphical representation for the selected at least one IoT device as an overlay on a display of the AR device; programming the interactive experience by generating a program for the behavior of the selected at least one IoT device using at least one of a logic gate or a control circuit connected to the selected at least one IoT device in a visual programming interface of the AR device, the logic gate and the control circuit together specifying the behavior to be performed by the IoT device in the visual programming interface; as well as A program that executes the behavior to selectively control the at least one IoT device during the interactive experience.

2. The method according to claim 1, further comprising: registering the at least one IoT device in response to registration information received from the at least one IoT device, the registration information comprising device input and output information for the at least one IoT device; as well as A type of AR user interface widget or graphical representation to be presented on the display for the at least one IoT device is determined from the registration information. 3 . The method of claim 1 , further comprising registering the AR device and receiving location information from the AR device.

4. The method according to claim 1, wherein: Programming the interactive experience by generating a program of behavior selected by the at least one IoT device includes accessing at least one of a logic gate or a control circuit through a visual programming interface of the AR device.

5. The method according to claim 1, wherein: The program generating the behavior of the selected at least one IoT device is based on input / output rules for the at least one IoT device, wherein the input / output rules are represented via Boolean logic, arithmetic functions, n-ary mappings, or mathematically definable operations.

6. The method according to claim 1, wherein: The procedure for generating the behavior of the selected at least one IoT device includes selecting the control circuit to express a function including at least one of a threshold, a clamping function, or an oscillator.

7. The method of claim 1 , further comprising representing at least one of a logic gate or a control circuit in a visual programming interface on a display of the AR device as a three-dimensional box having attachment points for input and output connections to other logic gates, control circuits, and IoT devices, the three-dimensional box including symbols describing the behavior represented by the three-dimensional box.

8. The method according to claim 1, wherein: Programming the interactive experience includes using a haptic device to employ force feedback when a logic gate or control circuit is adjusted or connected to another logic gate, control circuit, or IoT device.

9. The method according to claim 1, wherein: Programming the interactive experience includes generating a sequence of actions to be performed by the at least one IoT device during the interactive experience.

10. The method according to claim 9, wherein: Generating a sequence of actions to be performed by the at least one IoT device during the interactive experience includes sampling a state of the at least one IoT device and applying the sampled state to the timeline steps of the generated sequence of actions.

11. The method according to claim 1, wherein: Programming the interactive experience includes using connectors in the visual programming interface of the AR device to connect corresponding IoT devices, logic gates, and control circuits to specify program logic and information flow to be used in the interactive experience.

12. The method according to claim 1, wherein: Programming the interactive experience includes connecting corresponding IoT devices, logic gates, and control circuits by directly connecting the output port of one IoT device, logic gate, or control circuit to the input port of another IoT device, logic gate, or control circuit to specify the program logic and information flow to be used in the interactive experience.

13. The method according to claim 12, wherein: Connecting corresponding IoT devices, logic gates, and control circuits to specify program logic and information flow to be used in the interactive experience includes using at least one gateway node to logically connect corresponding IoT devices, logic gates, and control circuits that are separated by a certain distance in a display of the AR device.

14. The method according to claim 12, wherein: Connecting the respective IoT devices, logic gates, and control circuits to specify program logic and information flow to be used in the interactive experience includes using at least one IoT device proxy that represents the at least one IoT device at a location spatially separated from the at least one IoT device and exposes inputs and outputs of the at least one IoT device.

15. The method according to claim 1, wherein: Executing the program of behavior to selectively control the at least one IoT device during the interactive experience includes dividing execution of the program of behavior into discrete time steps and updating a state of the at least one IoT device during corresponding discrete time steps.

16. The method of claim 1, further comprising a program for debugging and replaying the behavior of the interactive experience via a debugging and replaying user interface of the AR device.

17. An augmented reality (AR) device for generating an interactive experience with at least one Internet of Things (IoT) device, comprising: monitor; User input devices; a runtime module that determines a type of AR user interface widget or graphical representation to be presented on the display for at least one IoT device, the runtime module receiving a selection of the at least one IoT device to be controlled during the interactive experience, and providing the determined type of AR user interface widget or graphical representation for the selected at least one IoT device to the display as an overlay; as well as an experience generation user interface that provides a visual programming interface through which a user provides input via the user input device to program the interactive experience by generating a program for the behavior of the selected at least one IoT device using at least one of logic gates or control circuits connected to the selected at least one IoT device in the visual programming interface, the logic gates and control circuits together specifying the behavior to be performed by the IoT device, Wherein, the runtime module executes the program of behavior to selectively control the at least one IoT device during the interactive experience programmed by the user.

18. The AR device according to claim 17 further includes a registration module, which registers the at least one IoT device in response to registration information received from the at least one IoT device, the registration information including device input and output information for the at least one IoT device, and the registration module determines the type of AR user interface widget or graphical representation to be presented on the display for the at least one IoT device from the registration information.

19. The AR device according to claim 17, wherein: The experience generation user interface is a builder component that includes a user interface component that includes at least one of the following: (a) an IoT device user interface that is used to interact with the at least one IoT device to attach logic gates or control circuits to inputs or outputs of the at least one IoT device; (b) a logic gate / control circuit user interface that provides access to logic gates for programming behavior based on input / output rules of the IoT device represented via Boolean logic, arithmetic functions, n-ary mappings, or mathematically definable operations, and access to control circuits to express functions including at least one of thresholds, clamping functions, or oscillators; or (c) a sequencer user interface that is used to define a sequence of actions performed by the at least one IoT device as part of the interactive experience.

20. A non-transitory computer-readable medium having instructions stored thereon, wherein: Execution of the instructions by one or more processors causes the one or more processors to interact with at least one Internet of Things (IoT) device using an augmented reality (AR) device to generate an interactive experience by performing operations including: receiving a selection of at least one IoT device to be controlled by the AR device during the interactive experience; providing, to the AR device, an AR user interface widget or graphical representation for the selected at least one IoT device as an overlay on a display of the AR device; programming the interactive experience by generating a program for the behavior of the selected at least one IoT device in a visual programming interface of the AR device using at least one of logic gates or control circuits connected to the selected at least one IoT device, the logic gates and control circuits together specifying the behavior to be performed by the IoT device; as well as A program that executes the behavior to selectively control the at least one IoT device during the interactive experience.

Images