Representation of a virtual reality object based on one or more conditions

Abstract

Headset (216), which includes the following: a case; a processor that is coupled to the chassis; a display (220) that is coupled to the housing and to which the processor can access, wherein the display (220) includes a display that is at least partially transparent, through which a wearer of the headset (216) can view objects of the real world and on which the headset (216) can display virtual reality objects; a first camera (222) that is directed away from the headset (216) to capture gestures and surroundings of the wearer and objects; a second camera that is aligned to image the wearer's eyes; a memory that is coupled to the chassis and can be accessed by the processor, the memory containing instructions that can be executed by the processor, for: Determine (304, 306) that at least one condition for displaying a virtual reality object in order to control a device other than the headset (216) is satisfied, wherein determining that at least one condition is satisfied includes determining that the wearer is looking at the device, based on the execution of eye tracking software using images from the second camera; Represent (308), in response to the determination, the virtual reality object on the display (220) such that it appears to the user to be arranged on an actual surface in three-dimensional space, wherein the actual surface is an unoccupied, substantially flat surface that is closest to the user, the nearest unoccupied flat surface being identified based on the execution of object recognition software using images from the first camera (222); and Sending (312) a command to the device to stop the operation of the device, the command being sent being based on a detection of an interaction of the wearer with the virtual reality object.

Description

AREA

[0001] The present application generally relates to a representation of a virtual reality object on a display based on one or more fulfilled conditions. STATE OF THE ART

[0002] As can be seen from the above, certain controls for control devices and applications in a dwelling or building may not always be located in a place convenient for the current user. This can frustrate the user.

[0003] German patent application JP 2013-172432A discloses a technique for detecting an operating device from video data, for displaying a virtual user interface for operating a detected operating device, and for inducing a user to operate the operating device in a virtual reality environment. German patent application DE 10 2015 208532A1 discloses an electronic device that detects a compatible external device, wherein the external device is running or has run a first application. The electronic device also receives usage information regarding the first application from the external device. An availability indicator for user selection is displayed based on the received information. Upon selection of the available application, a second application is launched on the electronic device, the second application being identical to the first.Patent CN 1 04 460 330 A discloses an augmented reality and remote control method based on indoor positioning and an electronic compass. Patent US 2014 / 0 247 208 A1 discloses an activation and waking of a computer device from a standby mode based on gaze detection. Patent US 2016 / 0 210 784 A1 discloses a portable, head-mounted display system comprising an eye-level display for showing an augmented reality object that is perceptible to the wearer of the head-mounted display system at an apparently real depth and in an apparently real location, and a controller for adjusting the apparent real location of the augmented reality object as a function of the wearer's field of view (FOV). Publication US 2015 / 0 193 018 A1 refers to the positioning of a target indicator via a display system using eye-tracking data.Publication US 6 232 972 B1 discloses a method for dynamically displaying controls in a toolbar. SUMMARY

[0004] The object of the present invention is to enable an improved display of a virtual reality object on a headset display.

[0005] This problem is solved by the subject matter of main claim 1 and dependent claim 10, which define the present invention.

[0006] Preferred embodiments of the present invention are the subject of the dependent claims.

[0007] According to one aspect, a device accordingly comprises a housing, a processor coupled to the housing, a display coupled to the housing and accessible to the processor, and memory coupled to the housing and accessible to the processor. The memory contains instructions that can be executed by the processor to determine that at least one condition for displaying a virtual reality object to control a device other than the device itself is met. The instructions can also be executed by the processor to display the virtual reality object on the display in response to the determination.

[0008] According to another aspect, a method involves determining that at least one condition for displaying a virtual reality object on a first device's display is met, and that the virtual reality object can be interacted with to control an output from a second device. The method also involves displaying the virtual reality object on the display in response to the determination, identifying a user's interaction with the virtual reality object, and controlling an output from the second device in response to this identification.

[0009] According to yet another aspect, a device comprises a first processor, a network adapter, and memory. The memory contains instructions that can be executed by a second processor to determine whether at least one condition for displaying a virtual reality object on a device's display, or for controlling a device other than the device itself, is met. The instructions can also be executed by the second processor to display the virtual reality object on the screen based on this determination. The first processor transmits the instructions to the second processor via the network adapter over a network.

[0010] The details of the present principles, both in terms of their structure and their operation, are best understood with reference to the accompanying drawings, in which identical reference numerals denote identical parts. These show: BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of an exemplary system according to the principles presented here. Fig. Figure 2 is an example block diagram of a network of devices according to the principles presented here. Fig. Figures 3 to 9 are flowcharts of exemplary logarithms according to the principles presented here. Fig. 10 and Fig. 11 are exemplary representations in accordance with the present principles and Fig. Figure 12 is an exemplary user interface (UI) according to the principles presented. DETAILED DESCRIPTION

[0011] The present invention relates to the use of a device for virtual reality (VR) and / or augmented reality (AR) in an augmented reality situation. Examples of such a device include a VR / AR headset, a device that has and uses a holographic display, a device that uses Google's Project Tango, and / or a device that uses 3D and / or indoor mapping technology to determine the dimensions of various areas and objects therein for VR / AR purposes, etc.

[0012] After displaying a virtual reality object in a location convenient for the user, the device can detect that a user is interacting with the object, for example, to control an internet-enabled device in the user's environment, such as a smart light. The object can be displayed based on one or more fulfilled conditions, so that objects the user is unlikely to want to interact with based on a given condition do not obstruct the user's field of vision, while objects that might be relevant to the user based on the given condition are displayed. For example, a virtual reality light switch for controlling spotlights on the user's property can be displayed when it is dark outside, but not when the sun is shining.

[0013] With regard to a computer system described herein, a system may include server and client components connected via a network, enabling the exchange of data between the client and server components. The client components may include one or more computing devices, including televisions (e.g., smart TVs or internet-enabled televisions), computers such as desktops, laptops, and tablets, so-called convertible devices (e.g., those with a laptop and tablet configuration), and other mobile devices, including smartphones. These client devices may, by way of non-limiting examples, use operating systems from Apple, Google, or Microsoft. The operating system may be Unix or a similar operating system, such as Linux.These operating systems can run one or more browsers, such as browsers made by Microsoft, Google, or Mozilla, or any other browser program that can access web pages and applications hosted by internet servers over a network, such as the internet, a local intranet, or a virtual private network.

[0014] As used herein, commands denote computer-implemented steps for processing information within the system. Commands can be implemented as software, firmware, or hardware, or combinations thereof, and can encompass any type of programmed step performed by components of the system. Thus, illustrative components, blocks, modules, circuits, and steps are sometimes described in terms of their functionality.

[0015] A processor can be any conventional general-purpose, single-chip, or multi-chip processor capable of executing logic using various lines, such as address lines, data lines, and control lines, as well as registers and shift registers. Furthermore, all logic blocks, modules, and circuits described herein can be implemented or executed using a general-purpose processor, a digital signal processor (DSP), a field-programmable gate array (FPGA), or any other programmable logic device, such as an application-specific integrated circuit (ASIC), discrete-gate or transistor logic, discrete hardware components, or a combination thereof, configured to perform the functions described herein. A processor can be implemented by a control device, a state machine, or a combination of computing devices.

[0016] Software modules and / or applications described herein by means of flowcharts and / or user interfaces may contain multiple subroutines, procedures, etc. Without limiting the disclosure, logic that is stated to be executed by a specific module may be redistributed to other software modules and / or combined into a single module and / or made available in a shared library.

[0017] Logic implemented in software can be written in a suitable language, such as C# or C++, and can be stored in or transmitted through a computer-readable storage medium (which is not, for example, a transitory signal), such as main memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CR-ROM), or other optical disk storage such as a Digital Versatile Disk (DVD), magnetic disk storage, or other magnetic storage devices, including USB flash drives, etc.

[0018] In one example, a processor can access information from data storage, such as a computer-readable storage medium, via its input lines, and / or the processor can wirelessly access information from an internet server by activating a wireless transceiver to send and receive data. When received, the data is typically converted from analog signals to digital by a circuit between the antenna and the processor's registers, and from digital to analog when transmitted. The processor then processes the data through its shift registers to computed output data on output lines for displaying the computed data on the device.

[0019] Components included in one embodiment can be used in any suitable combination in other embodiments. For example, any of the various components described herein and / or illustrated in the figures can be combined, exchanged, or excluded from other embodiments.

[0020] “A system with at least one of A, B and C” (likewise a “system with at least one of A, B or C” and a “system with at least one of A, B, C”) includes systems that include only A, only B, only C, A and B together, A and C together, B and C together and / or A, B, and C together, etc.

[0021] The term "circuit" or "circuit" may be used in the abstract, description, and / or claims. As is known in the field, the term "circuit" encompasses all levels of available integration, e.g., from discrete logic circuits to the highest level of circuit integration, such as VLSL, and includes programmable logic components programmed to perform the functions of an embodiment, as well as general-purpose or specialized processors programmed with instructions to execute these functions.

[0022] Now, specifically regarding Fig. Figure 1 shows an exemplary block diagram of an information handling system and / or a computer system 100. It should be noted that in some embodiments, the system 100 may be a desktop computer system, such as a ThinkCentre® or ThinkPad® PC, distributed by Lenovo (US) Inc. in Morrisville, North Carolina, or a workstation computer, such as the ThinkStation®, also distributed by Lenovo (US) Inc. in Morrisville, North Carolina. However, as can be seen from the present description, a client device, server, or other machine conforming to the present principles may include additional features or only some of the features of the system 100. Furthermore, the system 100 may, for example, be a game console, such as the Xbox®, and / or the system 100 may be a wireless phone, a notebook, a virtual / augmented reality headset, and / or another portable computing device.

[0023] As in Fig. As shown in Figure 1, the System 100 can include a so-called chipset 110. A chipset refers to a group of integrated circuits or chips configured to work together. Chipsets are usually sold as a single product (see, for example, chipsets sold under the brand names INTEL®, AMD®, etc.).

[0024] In the example of Fig. 1. The chipset 110 has a specific architecture that can vary to a certain extent depending on the brand or manufacturer. The architecture of the chipset 110 includes a core and memory control group 120 and an I / O control node 150, which exchange information (e.g., data, signals, commands, etc.) via, for example, a Direct Management Interface or a Direct Media Interface (DMI) 142 or a Link Controller 144. In the example of Fig. 1. The DMI 142 is a chip-to-chip interface (sometimes referred to as a connection between a "Northbridge" and a "Southbridge").

[0025] The core and memory control group 120 includes one or more processors 122 (e.g., single-core or multi-core, etc.) and a memory control node 126, which exchange information via a front-side bus (FSB) 124. As described herein, various components of the core and memory control group 120 can be integrated onto a single processor chip to, for example, create a chip that replaces the conventional "northbridge"-style architecture.

[0026] Memory control unit 126 connects to memory 140. For example, memory control unit 126 can provide support for DDR SDRAM memory (e.g., DDR, DDR2, DDR3, etc.). Generally, memory 140 is a type of main memory (RAM). This is often referred to as "system memory."

[0027] The memory control node 126 may also include a low-voltage differential signaling (LVDS) interface 132. The LVDS 132 may be a so-called LVDS display interface (LDI) for supporting a display device 192 (e.g., a CRT, a flat panel display, a projector, a touchscreen display, etc.). Block 138 includes some examples of technologies supported by the LVDS interface 132 (e.g., serial digital video, HDMI / DVI, display connector). The memory control node 126 also includes one or more PCI Express (PCI-E) interfaces 134, for example, to support discrete graphics 136. Discrete graphics using a PCI-E interface has become an alternative approach to an Accelerated Graphics Port (AGP). For example, the memory control node 126 can include a 16-lane (x16) PCI-E connector for an external PCI-E-based graphics card (including, for example, one of several GPUs).An example system might include AGP or PCI-E to support graphics.

[0028] In examples where it is used, the I / O control node 150 can include a variety of interfaces. The example of Fig. 1 includes a SATA interface 151, one or more PCI-E interfaces 152 (optionally one or more legacy PCI interfaces), one or more USB interfaces 153, a LAN interface 154 (more generally a network interface for communication over at least one network, such as the Internet, a WAN, a LAN, etc., under the leadership of the processor(s) 122), a general-purpose I / O interface (GPIO) 155, a low-pin-count interface (LPC) 170, a power management interface 161, a clock generator interface 162, an audio interface 163 (e.g., for speakers 194 for audio output), a total cost of ownership (TCO) interface 164, a system management bus interface (e.g., a serial multi-master computer bus interface) 165, and a serial peripheral flash memory / controller interface (SPI flash) 166, which in the example of Fig. It includes BIOS 168 and boot code 190. Regarding network connectivity, the I / O control node can include 150 integrated Gigabit Ethernet controller lines multiplexed with a PCI-E interface slot. Additional network features can function independently of a PCI-E interface.

[0029] The interfaces of the I / O control node 150 can facilitate communication with various devices, networks, etc. For example, the SATA interface 151, where used, enables reading, writing, or both reading and writing information to one or more drives 180, such as HDDs, SSDs, or a combination thereof. In any case, the drives 180 are to be understood as concrete, computer-readable storage media that are not transitory signals. The I / O control node 150 can also include an Advanced Host Controller Interface (AHCI) to support one or more drives 180. The PCI-E interface 152 enables wireless connections 182 to devices, networks, etc. The USB interface 153 is available for input devices 184, such as keyboards, mice, and various other devices (e.g., cameras, phones, storage devices, media players, etc.).

[0030] In the example of Fig. The LPC interface 170 enables the use of one or more ASICs 171, a Trusted Platform Module (TPM) 172, a Super I / O 173, a Firmware Node 174, BIOS support 175, and various types of memory 176, such as ROM 177, Flash 178, and non-volatile RAM (NVRAM) 179. Regarding the TPM 172, this module can be in the form of a chip that can be used to authenticate software and hardware devices. For example, a TPM can be capable of performing platform authentication and can be used to confirm that an access-seeking system is the expected system.

[0031] After powering on, System 100 can be configured to execute boot code 190 for BIOS 168, as stored in SPI flash memory 166, and then processes data under the control of one or more operating systems and application software (which is stored, for example, in system memory 140). An operating system can be stored in one of many locations and can be accessed according to the commands of BIOS 168.

[0032] Furthermore, the system 100 can include an audio receiver / microphone 191 which, based on a detected audio output, provides input from the microphone to the processor 122, such as from a user providing audible input / commands via the microphone 191. The system 100 can also include one or more cameras 193 that capture images and provide corresponding input to the processor 122.The cameras 193 can be thermal imaging cameras, digital cameras such as webcams, three-dimensional (3D) cameras and / or cameras that can be controlled by the processor 122 in other ways, and can be integrated into and / or arranged on the system 100 to capture images and / or videos, such as images of a user's eyes for eye tracking, images of gestures performed by the user, such as pointing at a specific object in the user's environment, and images of the environments themselves.

[0033] Additionally, although not shown for clarity, in some embodiments the system 100 may include a gyroscope that scans and / or measures the orientation of the system 100 and provides a corresponding input to the processor 122, as well as an accelerometer that scans any acceleration and / or movement of the system 100 and provides a corresponding input to the processor 122. Furthermore, and also not shown for clarity, the system 100 may include a GPS transceiver configured to receive geographic position information from at least one satellite and provide that information to the processor 122. It is understood, however, that according to the present principles, another suitable position receiver may be used instead of a GPS receiver to determine the position of the system 100.

[0034] It is understood that an example client device or other machine / computer may contain fewer or more features than System 100 according to Fig. Figure 1 shows that, in any case, it is understood, at least based on the foregoing, that System 100 is configured to execute the principles presented.

[0035] Now, with reference to Fig. Figure 2 shows exemplary devices that communicate over a 200 network, such as the Internet, according to the principles presented here. It is understood that each of the devices mentioned above... Fig. The devices described in section 2 may include at least some of the features, components and / or elements of the system 100 described above.

[0036] Fig. Figure 2 shows a notebook and / or convertible computer 202, a desktop computer 204, a portable device 206, such as a smartwatch, a smart TV (TV) 208, a smartphone 210, a tablet 212, a headset 216, a console 218, and a server 214, such as an internet server, that can provide cloud storage which devices 202-212, 216, and 218 can access. It is understood that devices 202-218 are configured to communicate over the network 200 in order to adopt the principles set forth herein.

[0037] To describe the headset 216 in more detail: It can include a housing to which a display device 220 is coupled for displaying virtual reality (VR) and / or augmented reality (AR) content. The display device 220 can include a display that is at least partially transparent, allowing the wearer of the headset 216 to view real-world objects and enabling the headset 216 to display virtual reality objects. The headset 216 can also include several cameras 222, which may be similar in function and configuration to the cameras 193 described above. At least one of the cameras is oriented to image the wearer's eyes when the wearer is wearing the headset 216, and another camera is oriented away from the headset 216 to image the wearer's gestures and surroundings, as well as objects.The headset 216 may also include a microphone 224, which may be similar to the microphone 191 in terms of function and configuration, a speaker 266, which may be similar to the speaker 194 in terms of function and configuration, and one or more head mounting elements 228 so that a user can attach the headset 216 to their head. Although not shown for clarity, it is understood that the headset 216 also includes a network interface for wired and / or wireless communication with the other devices. Fig. 2 can include, for example, the Internet, a local area network (LAN), a Bluetooth network, etc.

[0038] To describe Console 218 in more detail: It can be a virtual reality and / or augmented reality console for use in conjunction with Headset 216. For example, in addition to Headset 216, which has one or more cameras for tracking a user's eye movements, user gestures, and objects in the environment where Devices 216 and 218 are located, Console 218 can also include such cameras for the same or similar purposes. Console 218 can also operate in conjunction with Headset 216 to process data and execute the computer-implemented steps, commands, and functions described herein. Although not shown for clarity, it is understood that Console 218 also has a network interface for wired and / or wireless communication with the other devices. Fig. 2 can include, for example, the Internet, a local area network (LAN), a Bluetooth network, etc.

[0039] In Fig. Figure 3 is an example of an overall logic that can be executed by a device, such as System 100, Headset 216, Console 218, etc., to represent one or more virtual reality objects with which it can interact to control other devices according to the principles presented here. For convenience, the device that executes the logic of Fig. 3, referred to as a headset, and the logic described below can actually be executed wholly or partially by the headset in at least one embodiment.

[0040] Starting with block 300, the logic communicates with other devices on the network it uses, such as a password-protected Wi-Fi network or a Bluetooth network in a home environment. The logic can communicate with these other devices to receive status information, such as whether they are powered on or off, and / or other information regarding their operation.

[0041] From block 300, the logic can advance to block 302, where it can access a usage pattern record containing data about the days and times when one or more of the devices in the environment and / or the network-connected devices were controlled in the past, and even specific users / headset wearers who have controlled and / or connected one or more of the devices in the environment in the past. The record can be stored in cloud storage accessible to the headset, in memory within the headset itself, on another device that communicates with the headset over the network, and so on.

[0042] After block 302, the logic can advance to block 304, where the logic can monitor whether one or more conditions are met, which are determined based on the state information received in block 300 and / or on the record accessed in block 302.

[0043] From block 304, the logic can advance to a decision diamond 306, where it can determine whether one or more conditions are met. Specific conditions will be discussed further in relation to the Fig. As described in sections 4-9. A negative determination at rhombus 306 causes the logic to return to block 304 and / or continue with the determination at rhombus 306 until an affirmative determination is made.

[0044] In response to an affirmative determination at rhombus 306, the logic can proceed to block 308. At block 308, the logic can display a virtual reality object on the headset's display so that it appears to be positioned on a flat, unoccupied surface of the environment. The virtual reality object can be displayed in such a way that, regardless of the headset wearer's movement relative to the flat surface, it remains in the same position. This can be achieved using algorithms and / or processing that render three-dimensional (3D) graphics, algorithms and / or processing that render virtual reality graphics, algorithms and / or processing that render augmented reality graphics, and so on.Nevertheless, such algorithms and processing can be used according to the principles presented to display virtual reality objects even on non-flat surfaces, such as on the armrest of a couch where the headset wearer can sit.

[0045] From block 308, the logic can advance to decision rhombus 310. At rhombus 310, the logic can determine whether the wearer interacts with the virtual object, as determined based on input from one or more sensors, such as camera imaging gestures of the wearer, an inertial sensor scanning orientation and / or acceleration of the headset and / or another device worn by the wearer, a position transmitter / receiver on a device such as a smartwatch worn by the wearer while gesturing with the arm holding the smartwatch, etc.It is therefore understood that this sensor input can be used to determine whether a part of the user, such as their hand, is approaching or near a surface on which the virtual reality object appears to be located for the user, and whether that part of the user is moving in any direction in three-dimensional space in order to control the virtual reality object as if it were actually located on the physical surface in the three-dimensional space of the real world in which it appears to be located.

[0046] A negative determination at rhombus 310 causes the logic to continue making this determination until an affirmative determination is made. Then, in response to an affirmative determination at rhombus 310, the logic can advance to block 312, where it can send a command related to interacting with the virtual object to the device being controlled. The command can be to set an operation and / or output of the device being controlled, such as a light output for a lamp or a temperature output for an air conditioner.

[0047] Let's assume, for example, that the device to be controlled is a smart light or another smart application with an on / off switch. In this example, the virtual reality object can be a virtual on / off switch for the smart application. The wearer can interact with the virtual on / off switch by moving their hand to where the switch appears to be located and then moving their hand upwards to virtually toggle the switch, for example, from off to on. This gesture can be recognized by the headset through gesture recognition using images from the headset's camera or something else that maps the wearer's hand or arm. In response to the gesture recognition, the headset can issue a command to the smart light to adjust its light output by turning it on.

[0048] The Fig. 4-9 show examples of a logic that, in conjunction with Fig. 3 can be executed. For example, the decision at the corresponding decision diamonds in each of these figures can be made at decision diamond 306 as described above, and the action that takes place at each of the corresponding blocks in each of these figures can be carried out at block 308 as described above.

[0049] Starting with Fig. At block 400, the logic can determine whether the device to be controlled is switched on. In some examples, the decision might even be whether the device is switched on on a particular day and / or time when, according to the usage record being accessed, the device is typically switched off. "Typically" could be at least a threshold of the number of times on similar days / times in the past when the switching state most frequently occurred during a number of similar days / times, etc., in the recent past. A negative determination at block 400 can cause the logic to continue making this determination until an affirmative determination is made. Then, in response to an affirmative determination at block 400, the logic can advance to block 402, where the logic controls a virtual reality object (e.g., a virtual reality headset).a switch) which is pre-configured in the on position and can be interacted with to move it to the off position in order to turn off the device.

[0050] Fig. Figure 5 shows another example of logic. At block 500, the logic can determine whether the device to be controlled is switched off. In some examples, the decision can even be whether the device is switched off on a specific day and / or at a specific time when, according to the usage record being accessed, the device is typically switched on. A negative determination at block 500 can cause the logic to continue making this determination until an affirmative determination is made. Then, in response to an affirmative determination at block 500, the logic can advance to block 502, where the logic can represent a virtual reality object (e.g., a switch) that is preconfigured in the off position and can be interacted with to move it to the on position to switch the device on.

[0051] Fig. Figure 6 shows another example of the logic for use according to the present principles. At rhombus 600, the logic can determine whether the device to be controlled is switched on, even if no people are present in the room where the device is located. A person, or their absence, can be identified based on images from a camera in the room that maps the room and the execution of object and / or face recognition software on the images received from the camera to determine whether at least one person is present. A negative determination at rhombus 600 can cause the logic to continue with this determination until an affirmative determination is made. Then, in response to an affirmative determination at rhombus 600, the logic can advance to block 602, where the logic can identify a virtual reality object (e.g., a virtual reality camera).a switch) which is pre-configured in the on position and can be interacted with to move it to the off position to turn off the device, since there is no one in the room where the device is located, except for the headset wearer or anyone else.

[0052] With reference to Fig. The logic at rhombus 700 can determine whether a headset wearer is looking at and / or pointing at another device in the vicinity (e.g., with a finger or other part of the wearer's hand). The pointing gesture can be identified based on images from a camera capturing the wearer and by executing gesture recognition software on the received images to determine that the wearer is pointing at something. A wearer looking at the device can be identified based on images from a camera capturing the wearer's eyes and by executing eye-tracking software on the received images to determine where the wearer is looking. A negative determination at rhombus 700 can cause the logic to continue performing this determination until a positive determination is made.Then, in response to an affirmative determination at diamond 700, the logic can advance to block 702, where the logic can display at least one virtual reality object on a surface near the carrier in order to control the device using the virtual reality object. For example, if the device is a thermostat, virtual reality switches for turning a heater on or off and for turning an air conditioner on or off can be displayed on an adjacent table.

[0053] Now, with reference to Fig. At block 800, the logic can determine whether a command has been received that concerns another device in the vicinity, such as a voice command received via a headset microphone and recognized by speech recognition software. The command could be, for example, a command to display controls for the device and / or a command to control the device in a specific way. A negative determination at block 800 can cause the logic to continue making this determination until an affirmative determination is made. Then, in response to an affirmative determination at block 800, the logic can advance to block 802, where it can display at least one virtual reality object for controlling the device, such as an object associated with a specific function specified in the voice command and / or any objects with which the wearer wishes to interact to control the device.

[0054] With reference to Fig. At signal 9, the logic can determine whether a headset wearer is about to go to sleep. This determination can be based on input from a sleep sensor that scans one or more biometric data points of the wearer. Alternatively, it can be based on a recording associated with the wearer that shows an average time and / or various times the wearer has gone to sleep in the past. Thus, based on indicative data from the sleep sensor and / or based on an approaching time threshold that precedes the time the wearer is likely to go to sleep, the logic can determine whether the wearer is about to go to sleep based on the data in the recording. A negative determination at signal 900 can cause the logic to continue making this determination until a positive determination is received.Then, in response to an affirmative determination at diamond 900, the logic can advance to block 902, where it can display at least one virtual reality object for controlling one or more devices in other rooms of the same environment that are still switched on, such as a light in a common area or the kitchen of the user's personal residence. The virtual reality objects can be displayed in a location convenient for the user when they are about to go to bed, such as the top of a bedside table next to the user's bed.

[0055] Fig. Figure 10 is a representation of the principles at hand. A room 1000 has a floor 1002 and walls 1004 and 1006. A door 1008 is also shown in wall 1006, along with an actual real-world light switch 1010 that can be physically operated to turn an internet-enabled light 1012 on and off. It should be noted that the switch 1010 on wall 1006 is closest to the side of the door where the hinges 1014 are located, so that the switch 1010 may be obscured by the door swinging into room 1000 when the wearer of a headset enters room 1000 through door 1008, making it inconvenient for the wearer to reach the switch 1010 to turn on the light 1012.

[0056] Accordingly, based on a usage pattern of the wearer who has difficulty finding the light switch 1010 behind the door and in the dark (e.g., does not turn on the lights for at least a threshold time after entering the room 1000), the headset worn by the wearer can, after entering the room 1000 and / or opening the door 1008, represent a light switch 1016 of a virtual reality on its display, so that it appears to the wearer to be located on a flat section / area of ​​the wall 1006 next to the opposite side of the door, which is free of objects such as paintings, pictures, etc., that might otherwise be hanging on the wall.The user can thus enter room 1000 through door 1008 and conveniently interact with the light switch 1016 in a virtual reality by imitating a "switching on" motion at the point on wall 1006 where the virtual light switch 1016 is represented by the headset. The headset can then recognize the gesture and issue a command to switch on the light 1012.

[0057] Fig. Figure 11 is a further illustration of the principles presented. A room 1100 contains a television 1102, which is arranged in it together with a coffee table 1104 having a flat upper surface on which no real, physical objects are currently placed. However, for the wearer of a headset, who views the coffee table 1104 through the display of an augmented reality headset, one or more control sets 1106-1110 may appear arranged on the upper surface and in three-dimensional space, based on the headset identifying the surface as flat and unoccupied by other objects (as identified, for example, with the help of images from a camera together with object recognition software and / or spatial analysis software).

[0058] For example, object recognition can be used to determine that a real-world object identified in an image has a surface that the device, according to its preconfiguration, knows to be acceptable for displaying virtual reality objects. As another example, spatial analysis can be used to determine that a real-world object identified in an image has a surface that is at least substantially flat (e.g., a surface with less than the predefined curvature), and thus to determine that the surface is suitable for displaying virtual reality objects.

[0059] The control set 1106 can be used to control the television 1102, and accordingly, for the headset wearer, a transmitter up button 1112 and a transmitter down button 1114 may appear as if they were located on the upper surface of the coffee table 1104. Although only the buttons 1112 and 1114 for controlling the television are shown, other buttons for controlling the television may also be present, such as on and off buttons to turn the television on and off, respectively.

[0060] The control set 1108 can be used to control a light in another room and can therefore appear to the wearer of the headset as if it were located on the upper surface of the coffee table 1104.

[0061] The control set 1110 can be used to control a stove in a kitchen next to room 1100 and can accordingly appear to the wearer of the headset as if a temperature selector 1118 and an off switch 1120 were arranged on the upper surface of the coffee table 1104.

[0062] It is understood that the user can interact with each of the buttons / controls depicted as being arranged on table 1104 by gesturing to where they perceive the corresponding control / button to be located. Furthermore, it is understood that if actual real-world objects are arranged on part of the upper flat surface of coffee table 1104, the virtual reality objects can only be displayed as if they were arranged on other sections / areas of the upper surface where no objects have been placed.

[0063] Continuing with the detailed description regarding Fig. Figure 12 shows an exemplary user interface (UI) 1200, which can be displayed on a screen to configure device settings using the principles presented herein. UI 1200 includes an option 1202 to enable the display of a virtual reality object / control for one or more devices based on one or more conditions being met, as disclosed herein. Option 1202 can be enabled by selecting a checkbox 1204.

[0064] The UI 1200 also displays a sub-option 1206 for a user to select one or more specific devices 1208 for which controls can be displayed by checking the corresponding checkbox 1210 next to each device 1208. For simplicity, only a few devices 1208 are listed, but some and / or all controllable devices may be listed.

[0065] Furthermore, UI 1200 may include a sub-option 1212 for selecting one or more conditions of use according to the present principles by checking the corresponding checkbox 1216 next to each condition 1214. For simplicity, only a few conditions 1214 are listed; however, some and / or all conditions that can be monitored may be listed.

[0066] Furthermore, regarding Fig. 12. The UI 1200 can also include a setting 1218 for a user / carrier to preselect one or more actual surfaces in the real world within a specific environment, on which the controls of a virtual reality are to be displayed at the user's request. This allows the surfaces to be identified by a device using object recognition, based on the principles outlined above, prior to being displayed on the UI 1200. Each example surface can be selected by checking the corresponding adjacent checkbox 1220.

[0067] Furthermore, it should be noted that the first example surface, 1222, is not a single actual surface per se, but rather a type of actual surface. Specifically, the type is an unoccupied flat surface that is closest to a user at any given time. For example, if the user is sitting on a couch, the seat next to the user or the top of an adjacent coffee table might be the nearest surface, whereas if the user is leaning against a wall, the wall itself might be the nearest surface. The nearest surface can be identified based on images from a camera and the execution of object recognition and / or spatial analysis software, as well as based on positional data received from other devices connected to the headset.Whether an area is occupied or not can be determined based on the execution of object recognition software using images from a camera to identify the area and the objects that may be arranged on it.

[0068] Setting 1218 may also include a selector switch 1224. Selector switch 1224 can be selected to initiate a process in which a user, while wearing their headset, can walk around their environment and point to specific surfaces, which the headset then recognizes and assigns as surfaces on which virtual reality objects can be displayed according to the principles outlined.

[0069] It goes without saying that about Fig.12. In addition, further conditions can be monitored according to these principles to determine whether a virtual reality object is to be displayed. For example, weather and ambient light conditions can be used. If, for example, the light from the outside environment, which is sampled by an ambient light sensor, exceeds a threshold, then according to these principles a device cannot display an augmented reality light switch for turning on spotlights on a flat surface that would otherwise illuminate the outside environment.

[0070] To provide another example, if a user typically walks to the back door of their house after work in the dark, but the back door doesn't have an actual real-world light switch, the user's headset, when they let their dog out, could represent a virtual light switch alongside other real-world light switches on a wall next to the door. The user could then make a gesture to press or toggle the virtual light switch. In doing so, the headset could send a signal to the user's smart home system, which could then actually turn on the lights in the background. As another example, the user could add, configure, or instruct their headset (e.g.,to represent a virtual light switch for the spotlights at a specific location on the exterior wall of a house (permanently or whenever a predefined condition is met), provided that the spotlights are for exterior lighting.

[0071] It will now be clear that the principles presented here provide for the creation of a virtual object in appropriate areas of the user's virtual space / real world. By manipulating these virtual objects, a user can modify the function of objects in the real world. This can occur, for example, in a smart home environment, where users can electronically and remotely switch devices, applications, etc., on or off in their residence. Furthermore, in some implementations, the user can create switches, buttons, or any number of virtual objects to be displayed (and even assign specific locations where these are virtually displayed) in order to operate one of the connected devices.

[0072] Before concluding, it should be noted that although a software application implementing these principles may be sold with a device such as System 100, these principles apply only when such an application is downloaded from a server to a device via a network, such as the Internet. Furthermore, these principles apply when such an application is contained in a computer-readable storage medium that is sold and / or made available, provided that the computer-readable storage medium is not a transitory signal and / or a signal in itself.

[0073] It is understood that, although the present principles have been described with regard to some exemplary embodiments, these are not to be considered limiting and that various alternative arrangements can be used to implement the subject matter of the invention claimed herein. Components included in one embodiment can be used in any suitable combination in other embodiments. For example, some of the various embodiments described herein and / or illustrated in the figures can be combined, exchanged, or excluded from other embodiments.

Claims

[1] Headset (216), which includes the following: a case; a processor that is coupled to the chassis; a display (220) that is coupled to the housing and to which the processor can access, wherein the display (220) includes a display that is at least partially transparent, through which a wearer of the headset (216) can view objects of the real world and on which the headset (216) can display virtual reality objects; a first camera (222) that is directed away from the headset (216) to capture gestures and surroundings of the wearer and objects; a second camera that is aligned to image the wearer's eyes; a memory that is coupled to the chassis and can be accessed by the processor, the memory containing instructions that can be executed by the processor, for: Determine (304, 306) that at least one condition for displaying a virtual reality object in order to control a device other than the headset (216) is satisfied, wherein determining that at least one condition is satisfied includes determining that the wearer is looking at the device, based on the execution of eye tracking software using images from the second camera; Represent (308), in response to the determination, the virtual reality object on the display (220) such that it appears to the user to be arranged on an actual surface in three-dimensional space, wherein the actual surface is an unoccupied, substantially flat surface that is closest to the user, the nearest unoccupied flat surface being identified based on the execution of object recognition software using images from the first camera (222); and Sending (312) a command to the device to stop the operation of the device, the command being sent being based on a detection of an interaction of the wearer with the virtual reality object. [2] Headset (216) according to claim 1, wherein the interaction of the wearer with the virtual reality object is detected by determining, using images from the first camera (222), whether a part of the wearer is approaching the actual surface on which the virtual reality object appears to be arranged for the wearer, and whether the part of the wearer is moving in a direction in three-dimensional space to control the virtual reality object as if it were actually located on the actual surface in the three-dimensional space of the real world on which it appears to be arranged. [3] Headset (216) according to claim 1, wherein the instructions can be executed by the processor to: Determine that at least one condition is met, based on communication between the headset and the device. [4] Headset (216) according to claim 1, wherein the at least one condition comprises one of the following: that the device is switched on, that the device is switched off. [5] Headset (216) according to claim 1, wherein the at least one condition comprises that the device is switched on and that no one is in the room in which the device is arranged. [6] Headset (216) according to claim 1, wherein the at least one condition comprises a specific day and / or a specific time of day. [7] Headset (216) according to claim 1, wherein the condition comprises receiving a voice command from the wearer. [8] Headset (216) according to claim 1, wherein the condition comprises that the wearer points at the device. [9] Headset (216) according to claim 1, wherein the virtual object is a virtual switch for turning the device on and / or off. [10] Procedure (300), comprising the following: Determine (304, 306) that at least one condition is met for displaying a virtual reality object on a display (222) of a headset (216), wherein it is possible to interact with the virtual reality object to control an output from a device other than the headset (216), wherein the display (220) includes an at least partially transparent display through which a wearer of the headset (216) can view real-world objects and on which the headset (216) can display virtual reality objects, wherein the headset (216) comprises a first camera (222) and a second camera, wherein the first camera is oriented away (222) from the headset (216) to depict gestures and the environment of the wearer and objects, and the second camera is oriented to depict the wearer's eyes, wherein determining that at least one condition is met includes determining that the wearer is looking at the device.based on the execution of eye tracking software using images from the second camera; Represent (308), in response to the determination, the virtual reality object on the display such that it appears to the user to be arranged on an actual surface in three-dimensional space, wherein the actual surface is an unoccupied, substantially flat surface that is closest to the user, the nearest unoccupied flat surface being identified based on the execution of object recognition software using images from the first camera (222); and Sending (312) a command to the device to stop the operation of the device, the command being sent being based on a detection of an interaction of the wearer with the virtual reality object. [11] Method (300) according to claim 10, wherein the interaction of the carrier with the virtual reality object is detected by determining, using images from the first camera (222), whether a part of the carrier is approaching the actual surface on which the virtual reality object appears to be arranged for the carrier and whether the part of the carrier is moving in three-dimensional space in a direction to control the virtual reality object as if it were actually located on the actual surface in the three-dimensional space of the real world on which it appears to be arranged. [12] Method (300) according to claim 10, wherein the determination that the at least one condition is at least partially satisfied is carried out based on data relating to at least one usage pattern associated with the device. [13] Method (300) according to claim 10, wherein the virtual reality object is a button.

Images