Ultra-wideband radar device for controlling cloud-based gaming

Abstract

The wearable display of the client system is configured to receive and display a game stream representing a gaming application executing on a cloud-based gaming server. To aid in controlling the gaming application, the client system includes or is otherwise connected to a smart ecosystem including one or more ecosystem devices each including radar circuitry. In response to the wearable display displaying the game stream, the ecosystem devices are configured to transmit one or more radar signals toward the wearable display and generate radar data. Based on the radar data, the client system then determines motion data for the wearable display, the user, or both, and generates one or more game inputs for the gaming application based on the determined motion data.

Description

[Background technology]

[0001] background Traditionally, virtual reality video gaming applications run on local hardware to create a virtual environment that is provided to a virtual reality headset worn by a user. To help ensure that the virtual environment of the virtual reality gaming application moves as the user moves, many virtual reality systems include one or more virtual reality base stations that must be placed around the user. These virtual reality base stations use laser-based optical sensors to determine the location and movement of the virtual reality headset and the user. The real video gaming application then modifies the virtual environment based on the location and movement so that the virtual environment changes as the user and virtual reality headset move. However, placing these virtual reality base stations around the user reduces the area in which the user can move, negatively impacting the user experience. Furthermore, using a dedicated virtual reality base station increases the number of elements required for the virtual reality system, making the setup of the virtual reality system more cumbersome. Additionally, to help ensure that the virtual environment of the virtual reality gaming application moves as the user moves, some virtual reality systems include one or more cameras integrated into the virtual reality headset that are used to determine the location and movement of the virtual reality headset and the user. However, such cameras require high power to function, increasing the power consumption of the system.

[0002] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the following drawings, in which: The use of the same reference symbols in different drawings indicates similar or identical items. [Brief description of the drawings]

[0003] [Figure 1]FIG. 1 is a block diagram of a cloud-based gaming system for single player or multiplayer (including massively multiplayer) gaming according to some embodiments. [Diagram 2] FIG. 1 is a block diagram of a client system configured to implement a radar for cloud-based gaming input according to some embodiments. [Diagram 3] FIG. 13 is a signal flow diagram for an example process for implementing radar for cloud-based gaming, according to some embodiments. [Figure 4] A block diagram of an example processing device 400 for implementing a radar for cloud-based gaming input is presented, according to some embodiments. [Diagram 5] FIG. 1 is a flow diagram of an example method for implementing a radar for cloud-based gaming input according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Overview of the embodiment According to an example embodiment, a method for cloud-based gaming may include transmitting a radar signal in response to a wearable display receiving a game stream. The method may further include determining movement data of a user of the wearable display from echoes of the radar signal, and generating one or more game inputs based on the movement data of the user of the wearable display.

[0005] The exemplary method may further include modifying the game stream based on one or more game inputs and transmitting the modified game stream to the wearable display. In addition, the method may include generating sensor data associated with the user at the wearable display, where the one or more game inputs are further determined based on the sensor data associated with the user. Furthermore, the method may include receiving one or more interactions at the input device, where the one or more game inputs are further determined based on the input data representing the one or more interactions. The method may also include transmitting the one or more game inputs to a cloud-based server associated with the game stream. The motion data may indicate a location of the user, a location of the wearable display, or both. Furthermore, the wearable display may be connected to a smart ecosystem including an ecosystem device configured to transmit a radar signal. Also, the ecosystem device may include at least one of a smart hub, a display, or a sensor. Additionally, the radar signal may include an ultra-wideband radar signal.

[0006] In another example embodiment, a method for cloud-based gaming may include receiving, in response to initiating a client gaming session, sensor data associated with a user of the wearable display from a wearable display and radar data associated with the user of the wearable display from an ecosystem device. In addition, the method may include determining one or more game inputs based on the sensor data and the radar data associated with the user of the wearable display. The method may also include receiving a modified game stream based on the one or more game inputs in response to transmitting the one or more game inputs to a cloud-based server associated with the client gaming session.

[0007] The method may further include generating a game stream at a cloud-based server associated with the client gaming session and modifying the game stream based on one or more game inputs. Also, the wearable display and ecosystem device may be in a smart ecosystem. The ecosystem device may include radar circuitry configured to transmit an ultra-wideband radar signal.

[0008] Additionally, the method may include receiving input data from the input device representing one or more interactions with the input device, where the one or more game inputs are further determined based on the input data. Furthermore, the method may include transmitting a radar signal and, in response to receiving an echo of the radar signal, determining radar data associated with a user of the wearable display based on the echo of the radar signal. Within the method, the ecosystem device may include at least one of a smart hub, a display, or a sensor. The method may further include determining motion data associated with a user of the wearable display based on the radar data, where the one or more game inputs are further determined based on the motion data. In the method, the motion data may indicate a position of the user of the wearable display, a position of the wearable display, or both.

[0009] In an exemplary embodiment, a cloud-based gaming system includes one or more cloud-based gaming servers communicatively coupled to one or more client systems. The client systems may each include a wearable display including one or more processors, and one or more ecosystem devices. At least one ecosystem device may include one or more processors and a memory coupled to the one or more processors that stores executable instructions configured to operate the one or more processors to perform any of the methods disclosed herein.

[0010] In another exemplary embodiment, a cloud-based gaming system includes one or more cloud-based gaming servers communicatively coupled to one or more smart ecosystems. The smart ecosystems may each include a wearable display including one or more processors, and one or more ecosystem devices. At least one ecosystem device may include one or more processors and a memory coupled to the one or more processors that stores executable instructions configured to operate the one or more processors to perform any of the methods disclosed herein.

[0011] Detailed Description Within a cloud-based gaming system, a client system is configured to receive a stream of video game frames representing visual content of a video game instance running on its server or an associated server, and display the stream of video game frames on one or more devices. For example, some client systems include a virtual reality headset (e.g., a wearable display) configured to display the stream of video game frames. To control a game represented by the stream of video game frames displayed by such a virtual reality headset, the client system further includes an input device configured to receive an interaction from a user, such as a gamepad, a joystick, a button, a keyboard, a computer mouse, a touch screen, or any combination thereof. In response to receiving the interaction at the input device, the client system is configured to transmit data representing the interaction to a server running a video game instance associated with the stream of video game frames displayed by the virtual reality headset. As an example, the client system determines a game input (e.g., a signal interpretable by the video game instance as one or more commands) based on the interaction at the input device, and transmits the game input to the associated server. After receiving data representing input (e.g., game input), the server provides such data as input to the video game instance to modify the video game instance and the stream of video game frames provided to the virtual reality headset. In this manner, the client system controls the game represented by the stream of video game frames displayed by the virtual reality headset.

[0012] In addition, to aid in the control of a game represented by a stream of video game frames displayed by a virtual reality headset (e.g., a wearable display), the virtual reality headset includes one or more sensors configured to generate data indicative of one or more movements of a user. For example, the virtual reality headset includes sensors configured to track the user's head movements, virtual reality movements, or both. After such data is generated, the client system determines game inputs based on the indicated user movements and transmits the game inputs to an associated server (e.g., a server running the video game instance). In some embodiments, the client system further includes a virtual reality base station to aid in determining the user's movements (e.g., the user's head movements, the user's body movements, the virtual reality headset movements, the input device movements). Such a virtual reality base station may include, for example, one or more laser-based tracking systems and be configured to be positioned in proximity to the user such that the user's movements may be determined. However, positioning such a virtual reality base station in proximity to the user may limit the area in which the user may move, adversely affecting the user experience. In addition, using a dedicated virtual reality base station increases the number of elements required for a virtual reality system, making the virtual reality system more cumbersome to set up, negatively impacting the user experience, and increasing costs. Furthermore, in some embodiments, the virtual reality headset includes one or more cameras mounted within the virtual reality headset that track the user's head movements, the virtual reality movements, or both. However, such cameras mounted within the virtual reality headset increase the power consumption of the virtual reality headset, increasing the cost of operating the virtual reality headset.

[0013] To this end, the techniques and systems described herein are directed to implementing ultra-wideband (UWB) radar for cloud-based gaming input. For example, to aid in determining a user's movements within a client system including a virtual reality headset (e.g., a wearable display), the client system includes ecosystem devices each implementing radar circuitry (e.g., UWB radar circuitry) configured to transmit one or more radar signals (e.g., UWB radar signals) and receive one or more echoes of the transmitted radar signals. These ecosystem devices include, for example, smart hubs, virtual assistant devices, computable televisions (i.e., smart TVs), sensors (e.g., smoke detectors, air sensors, thermostats), computable lighting (smart lighting), drones, routers, modems, Internet of Things (IoT) devices, or any combination thereof. In some embodiments, such ecosystem devices also include virtual reality headsets, input devices, or both, or are otherwise included within a smart ecosystem (e.g., a group of devices that communicate with each other using one or more communication protocols) connected to them (e.g., via a local area network (LAN)). In response to the virtual reality headset receiving a stream of video game frames (e.g., from a cloud-based gaming server), radar circuitry in one or more ecosystem devices is configured to transmit one or more radar signals (e.g., UWB radar signals) toward a user of the virtual reality headset. These radar signals, for example, have a predetermined frequency (e.g., a frequency between 3.1 GHz and 10.6 GHz) and are configured to bounce off one or more objects (e.g., a user of the virtual reality headset, a virtual reality headset, an input device) and produce one or more echoes (e.g., bounced radar signals) that are received by the radar circuitry.

[0014] Based on the echoes, the client system determines user motion data representing the user's position, the movement of the user's body (e.g., torso, legs, arms, fingers), the movement of the user's head, the movement of the virtual reality headset, the movement of the input device, or any combination thereof. For example, based on receiving radar data representing the received echoes, the client system determines the time of flight of the echoes, and based on the time of flight, determines the position and movement of the user's head. The client system then determines one or more game inputs based on the motion data and provides the game inputs to an associated server. In this manner, the client system uses ultra-wideband (UWB) radar to determine the user's movement without the need for a virtual reality base station, increasing the area in which the user can move and improving the user experience. Furthermore, using ecosystem devices that also have radar circuitry for determining the user's movement instead of a dedicated virtual reality base station reduces the number of elements required for the virtual reality system, simplifying the setup of the virtual reality system and improving the user experience.

[0015] The techniques of this disclosure are described in the example context of a cloud gaming system, in which a cloud-based server or other remote server renders a stream of video frames representing the visual content of a video game instance running on that server or an associated server, and then encodes each frame and generates a bitstream representing the stream of encoded and rendered game frames for transmission over one or more networks to one or more client systems. However, it will be understood that the systems and techniques described herein are not limited to this example context, but may instead be implemented in any of a variety of video stream encoding / decoding systems using the guidelines provided herein.

[0016] 1 illustrates a cloud-based gaming system 100 for single-player or multiplayer (including massively multiplayer) gaming, according to some embodiments. The cloud-based gaming system 100 includes one or more cloud-based gaming servers 102, each of which is communicatively coupled to one or more client systems 112 by at least one network 110. The network 110 is configured to enable the transmission and reception of data between any of the servers 102 and the client systems 112, and includes, for example, wired and wireless networks, such as Ethernet, Internet, Wi-Fi, or any combination thereof. In embodiments, each of the servers 102 are communicatively interconnected via a high-bandwidth, low-latency inter-server messaging bus. The servers 102 are typically distributed in one or more data centers across a geographic region to reduce transmission latency through physical proximity. In the illustrated embodiment, three servers 102-1, 102-2, and 102-3 are presented supporting four client systems 112-1, 112-2, 112-3, and 112-4, although in other embodiments any number of servers may be implemented to support any number of client devices. It will be understood that in a typical real-world implementation, the quantity of servers 102 and the quantity of client systems 112 will typically be significantly greater than that presented in the example of FIG.

[0017] In the depicted embodiment, each server 102 includes a computing device, e.g., a cloud server, a virtual server, or any combination thereof, configured to support one or more client gaming sessions executed on one or more client systems 112. A "client gaming session," as used herein, includes a gaming application being played, partially simulated, or fully simulated on a client system 112. Each server 102 is configured to support this client gaming session by executing a corresponding gaming platform instance 104 that facilitates the execution of at least a portion of the gaming application being played, partially simulated, or fully simulated on the client system 112. Such facilitation may include performing one or more operations related to the gaming application, e.g., rendering one or more frames related to the gaming application, performing computations related to the gaming application (e.g., lighting computations, artificial intelligence computations, physics computations, shading computations, input computations, etc.), providing access to files, or any combination thereof, to name a few. The gaming platform instance 104 provides various software and hardware resources to accomplish such facilitation, such as communication / network management, resource management, media rendering encoding, etc. In this manner, the gaming platform instance 104 simulates the execution of one or more operations of a gaming application for a corresponding player as if the gaming application were being played on a local gaming device, such as a personal computer ("PC"), game console, smartphone, tablet computer, automobile entertainment system, etc.

[0018] Each client system 112 represents hardware and software resources utilized to receive player input through the operation of one or more input / output devices of at least one player, and to present video and audio content that respectively represent the visual and aural content of the gameplay of at least one player. Examples of client systems 112 include one or more desktop computers, notebook computers, tablet computers, virtual reality systems, augmented reality systems, computing-enabled cellular phones (i.e., "smartphones"), computing-enabled televisions (i.e., "smart televisions"), or any combination thereof, to name a few. As shown with reference to client system 112-1, each client system 112 includes one or more client devices 116. In the illustrated embodiment, client system 112-1 comprises a first client device 116-1 that is communicatively coupled to or otherwise associated with a display 118, at least one input device 120 (e.g., gamepad, joystick, keyboard, mouse, touch screen), one or more network interfaces configured to couple the client system 112 to a network connecting a corresponding server 102, one or more processors, memory, storage, speakers, and other computing resources for rendering, processing, and displaying scenes of the virtual environment.

[0019] According to embodiments, one or more client devices 116 include ecosystem devices in a smart ecosystem. Such a smart ecosystem, for example, represents a group of ecosystem devices each having respective capabilities and running respective applications associated with those capabilities. In a smart ecosystem, the ecosystem devices are networked with each other using one or more communication protocols (e.g., ZigBee, Bluetooth, Bluetooth Low Energy (LE), Wi-Fi) to provide their respective capabilities to one or more users. These ecosystem devices include, for example, smart hubs, virtual assistant devices, smart TVs, sensors (e.g., smoke detectors, air sensors, thermostats), smart lighting, drones, routers, modems, user devices (e.g., laptop computers, tablet computers, computing-enabled wearable devices), IoT devices, or any combination thereof. In some embodiments, the smart ecosystem further includes or is otherwise communicatively coupled to (e.g., via one or more communication protocols) one or more client devices 116 (e.g., virtual reality headset (e.g., wearable display) 116-3), input device 120, or both. According to embodiments, the one or more ecosystem devices include radar circuitry including one or more transceivers, transmitters, receivers, antennas, or any combination thereof, configured to generate and transmit one or more radar signals (e.g., UWB radar signals) having one or more predetermined frequencies. For example, the ecosystem devices are configured to transmit and generate one or more radar signals having frequencies between 3.1 GHz and 10.6 GHz. The transmitted radar signals are configured to bounce off one or more objects (e.g., a user, a virtual reality headset 116-3, an input device 120) and create one or more echoes.In response to receiving the echo, the radar circuitry of the ecosystem device is configured to determine radar data indicative of the signal's time of flight (e.g., the time between transmission of the radar signal and receipt of the echo), the angle of the received echo, and the angle of the transmitted radar signal, or any combination thereof.

[0020] As shown with reference to client system 112-1, each client system 112 may include a decoder 114 configured to decode one or more frames associated with a virtual environment. The decoder 114 may include hardware and software configured to decode one or more encoded streams (e.g., game stream 108) received from server 102 to produce a decoded frameset or stream. Each decoder 114 is configured to decode any encoded frame encoded by any method or process disclosed herein. In embodiments, client system 112-1 further includes a smartphone client device 116-2 and a wearable virtual reality client device 116-3, each of which may operate as an integrated mobile computing device having input devices, output devices, display devices, and communication devices similar to those mentioned with respect to client system 112-1 above.

[0021] Although certain aspects described herein are described with specific reference to a cloud gaming scenario, it will be understood that in certain embodiments, the described techniques may be utilized in various non-gaming scenarios, such as, for example, when one or more of the server 102 and client systems 112 operate to render, process, and display other types of informational content, educational content, entertainment content, specialized content, artistic content, or any combination thereof. Thus, while the techniques are described herein with respect to rendering content that may utilize specific examples related to cloud gaming and gaming content, it will be further understood that such description and techniques may be applied to such non-gaming scenarios. Although the examples provided herein may refer to scenarios involving the rendering, processing, and display of such content due to the particular bandwidth and network latency challenges associated with gaming content, they should not be construed as indicating that the described techniques are limited to those scenarios.

[0022] In operation, each server 102 executes a gaming platform instance 104 for one or more client gaming sessions. Executing the gaming platform instance 104 includes rendering a set of gaming frames including one or more gaming frames associated with a gaming application executing on one or more respective client systems 112. Each rendered gaming frame depicts at least a portion of a virtual environment used within the gaming application executing on the client system 112. For example, each rendered gaming frame may depict at least a portion of a virtual environment displayed on a display 118 of the client system 112 during a client gaming session.

[0023] Each server 102 is configured to encode each rendered gaming frame via a respective encoder 106 to generate a respective encoded frameset (also referred to herein as a “game stream” 108). Each server 102 is configured to encode the game stream 108, for example, through compression, reorganization, and manipulation of each frame rendered by the gaming platform instance 104. In embodiments, the encoder 106 of a server 102 implements one or more codecs to encode one or more rendered frames in accordance with one or more codecs. Such codecs may include H.264, H.265, VP9, ​​AV1, or any combination thereof, to name a few. Each resulting game stream 108 corresponds to a gaming application running on one or more client systems 112 and is provided to those corresponding client systems 112 over the network 110. Each corresponding client system 112 is configured to decode the received game stream 108 via a decoder 114 and display the resulting decoded set of frames 122, for example, on a display 118 (e.g., a display 118 included in a virtual reality headset 116-3). Each client system 112 is configured to decode the respective game stream 108 by compressing, reorganizing, and manipulating frames in the encoded stream according to one or more various video codecs, including lossless and lossy codecs. According to embodiments, each client system 112 includes a decoder that implements one or more codecs to decode the received game stream 108 according to the one or more codecs. Such codecs may include H.264, H.265, VP9, ​​AV1, or any combination thereof, to name a few.In the illustrated embodiment, three game streams 108-1, 108-2, 108-3 are depicted, however, in other embodiments, the server 102 may generate any number of game streams 108, each corresponding to one or more client gaming sessions.

[0024] In embodiments, the client system 112 includes a virtual reality headset (e.g., wearable display) 116-3 configured to be worn by a user and to display the decoded frameset 122 (e.g., an encoded game stream) to the user. To this end, in embodiments, the virtual reality headset 116-3 includes, for example, a display 118. To control a gaming application associated with the decoded frameset 122 (e.g., a gaming application executed to generate the decoded frameset 122) displayed on the virtual reality headset 116-3, the client system includes one or more input devices 120 (e.g., a keyboard, a gamepad, a joystick, a virtual reality controller, a computer mouse, a touch screen, a smartphone, a microphone) each configured to receive one or more interactions from the user. Such interactions include, for example, gestures, button clicks, keystrokes, drags, rotations, movements, voice commands, noises, or any combination thereof. In response to receiving one or more interactions, the input devices 120 are configured to transmit data indicative of the interactions to a processing device within the client system 112. Such processing devices may include, for example, one or more processors each included within or otherwise connected to a respective client device (e.g., computer, smartphone, ecosystem device, virtual reality headset) of client systems 112. According to embodiments, after receiving data indicative of an interaction, the processing devices are configured to generate one or more game inputs based on the interaction. These game inputs may each include, for example, signals that can be interpreted as one or more commands by a gaming application associated with the game stream 108 displayed on the virtual reality headset 116-3.To generate the game inputs, for example, the processing device compares one or more interactions indicated in the received data to data in one or more tables, arrays, vectors, libraries, or any combination thereof to determine one or more game inputs. After determining the one or more game inputs, the processing device transmits the game inputs to the server 102 executing a gaming application associated with the decoded frameset 122 (e.g., a decoded game stream) to be displayed on the virtual reality headset 116-3. In response to receiving data representing the one or more interactions, the one or more game inputs, or both, the server 102 is configured to modify the gaming application by generating one or more game inputs, providing one or more game inputs to the gaming application, or both. For example, the server 102 provides the gaming application with one or more received game inputs that modify the gaming application (e.g., the game inputs are used as inputs to the gaming application). Once the gaming application is modified, the server 102 generates a game stream 108 (e.g., a modified game stream) based on the modified gaming application and transmits the game stream 108 to the client system 112.

[0025] To further aid in controlling a gaming application associated with the decoded frameset 122 displayed on the virtual reality headset 116-3, the virtual reality headset 116-3 includes one or more sensors configured to generate sensor data indicative of a position of a user (e.g., a user wearing the virtual reality headset 116-3), a position of the virtual reality headset 116-3, a position of one or more input devices 120 (e.g., an input device held by a user), a movement of the user (e.g., direction, speed, acceleration), a movement of the virtual reality headset, a movement of the one or more input devices, or any combination thereof. These sensors may include, for example, one or more optical sensors (e.g., photoelectric sensors, laser-based sensors, infrared sensors), ultrasonic sensors, Hall effect sensors, UWB radar systems, or any combination thereof. In various embodiments, the sensors of the virtual reality headset 116-3 are configured to transmit the sensor data to a processing device (e.g., a computer, a smartphone, a virtual reality headset, an ecosystem device) of the client system 112. In response to receiving such sensor data, the processing device is configured to determine motion data based on the received sensor data. Such motion data may indicate, for example, a position of a user, a position of the virtual reality headset 116-3, a position of one or more input devices, a movement (e.g., direction, speed, acceleration) of the user, a movement of the virtual reality headset, a movement of one or more input devices, or any combination thereof. Based on the determined motion data, the processing device then determines one or more game inputs, for example, by comparing the motion data to data in one or more tables, arrays, vectors, libraries, or any combination thereof. In various embodiments, the processing device transmits the motion data, the one or more game inputs, or both to a server 102 associated with the decoded frameset 122 to be displayed on the virtual reality headset 116-3 (e.g., a server executing a gaming application associated with the decoded frameset 122).The server 102 then modifies the gaming application by generating one or more game inputs, providing one or more game inputs to the gaming application, or both. After modifying the gaming application, the server 102 transmits the modified game stream 108 to the client system 112.

[0026] Additionally, to aid in controlling a gaming application associated with the decoded frameset 122 displayed on the virtual reality headset 116-3, the client system 112 includes one or more ecosystem devices each including radar circuitry (e.g., UWB circuitry) configured to generate and transmit one or more radar signals. For example, the client system 112 includes an ecosystem device configured to generate and transmit one or more radar signals in the direction of the virtual reality headset 116-3. After transmitting the one or more radar signals, the radar circuitry of the ecosystem device is configured to receive one or more echoes based on the transmitted radar signals. The ecosystem device then transmits radar data indicative of the transmitted radar signals, the received echoes, or both, to a processing device (e.g., a computer, a smartphone, a virtual reality headset, an ecosystem device) within the client system 112. Based on the radar data, the processing device is configured to determine motion data of the virtual reality headset 116-3, the user, the input device 120 (e.g., the input device 120 held by the user), or any combination. For example, the processing device is configured to determine one or more of a time of flight, a transmission angle, a reception angle, or any combination thereof based on the received radar data. The processing device then determines motion data (e.g., a user's position and movement) based on the determined time of flight, transmission angle, and reception angle. In embodiments, the processing device then determines one or more game inputs based on the motion data, for example, by comparing the motion data to data in one or more tables, arrays, vectors, libraries, or any combination thereof, and transmits the game inputs to a server 102 associated with the game stream 108 that modifies an associated gaming application. In this manner, the client system 112 determines the user's movement using radar (e.g., UWB radar) without requiring a virtual reality base station to increase the amount of free zone around the user.Additionally, using ecosystem devices within the client system 112 to determine the movement of the virtual reality headset 116-3 and the user instead of a dedicated virtual reality base station reduces the number of elements required in the virtual reality system and improves the user experience.

[0027] 2, a client system 200 configured to implement a radar for cloud-based gaming input is presented. In embodiments, the client system 200 includes a smart ecosystem 232, similar to or equivalent to the client system 112, that represents a group of devices (ecosystem devices 228) each having respective capabilities and networked together using one or more communication protocols (e.g., ZigBee, Bluetooth, Bluetooth LE, Wi-Fi) to provide their respective capabilities to one or more users. For example, the smart ecosystem 232 includes one or more ecosystem devices 228 each communicatively coupled to one or more other ecosystem devices 228 by a local area network (LAN) 222 (e.g., wireless network, Wi-Fi network). 2 presents a smart ecosystem 232 having three ecosystem devices (228-1, 228-2, 228-N) representing the N ecosystem devices 228, in other embodiments, the smart ecosystem 232 can have any number of ecosystem devices 228. Such ecosystem devices 232 include, for example, smoke detectors, air sensors, thermostats), smart lighting, drones, routers, modems, IoT devices, or any combination thereof. In some embodiments, the smart ecosystem 232 further includes or is otherwise communicatively coupled to (e.g., via the LAN network 222) a virtual reality headset 116-3 (e.g., a wearable display), one or more input devices 120 (e.g., a virtual reality controller), or both.

[0028] According to embodiments, one or more devices (e.g., virtual reality headset 116-3, input device 120, ecosystem device 228) in smart ecosystem 232 are communicatively coupled to network 110 via LAN network 222 to receive a game stream 108 representing game frames from a gaming application executing on server 102. According to embodiments, client system 200 includes a decoder (e.g., decoder 114) configured to decode the received game stream 108 and provide the decoded game stream 108 to virtual reality headset (e.g., wearable display) 116-3 for display. For example, client system 200 includes one or more client devices 116 (e.g., computer, gaming console, smartphone, virtual reality headset, ecosystem device) that includes a decoder configured to decode the received game stream 108 and provide a decoded set of frames 122 to virtual reality headset 116-3. To control a gaming application represented by the decoded game stream displayed by the virtual reality headset 116-3, the client system 200 includes one or more input devices 120. For example, the client system 200 includes one or more virtual reality controllers, each including an accelerometer, a motion sensor, a button, a joystick, a touchpad, a microphone, or any combination thereof. While the decoded game stream is displayed by the virtual reality headset 116-3, the one or more input devices 120 are configured to receive one or more interactions (e.g., gestures, button clicks, keystrokes, drags, rotations, movements, voice commands, noises) from a user (e.g., a user of the virtual reality headset 116-3). In response to receiving the one or more interactions, the input device 120 transmits data representing the interaction to a processing device within the client system 200.Such a processing device (not shown for clarity) may, for example, include one or more processors each included within or otherwise connected to a respective client device 116 (e.g., computer, smartphone, ecosystem device, virtual reality headset) of client system 200. For example, the processing device may include one or more processors included within virtual reality headset 116-3. Based on the interaction (e.g., based on input data indicative of the interaction), the processing device determines one or more game inputs, for example, by comparing the interaction to data in one or more tables, arrays, vectors, libraries, or any combination thereof. The processing device then transmits data representing the interaction, the one or more game inputs, or both, to a server 102 executing a gaming application represented by a decoded game stream displayed on virtual reality headset 116-3. The server 102 then modifies the gaming application based on the received data representing the interaction, the one or more game inputs, or both, for example, by providing the one or more game inputs to the gaming application. In this manner, the client system 200 uses the input device 120 to control (eg, modify) a gaming application executing on the server 102 .

[0029] To further aid in control of a gaming application represented by the decoded game stream displayed by the virtual reality headset 116-3, the virtual reality headset 116-3 includes one or more sensors configured to generate sensor data indicative of a position, movement (e.g., direction, speed, acceleration), or both, of the user, the virtual reality headset 116-3, one or more input devices 120 (e.g., input device 120 held by the user), or any combination thereof. For example, the virtual reality headset 116-3 includes an accelerometer configured to generate sensor data indicative of a direction of the user. Based on the sensor data, the processing device of the client system 200 is then configured to determine motion data (e.g., a position, movement, or both, of the user, the virtual reality headset 116-3, one or more input devices 120), of the user, the virtual reality headset 116-3, the input device 120, or any combination thereof. For example, the computer of the client system 200 includes one or more processors configured to determine a position and movement of the virtual reality headset 116-3. Using such motion data, the processing device then determines one or more game inputs, for example, by comparing the motion data to data in one or more tables, arrays, vectors, libraries, or any combination thereof. In embodiments, the processing device then transmits the motion data, the game inputs, or both to a server 102 executing a game application associated with the displayed game stream 108 to modify the game application.

[0030] Additionally, according to embodiments, one or more ecosystem devices 228 include radar circuitry 230 including hardware-based circuitry, software-based circuitry, or both configured to transmit one or more radar signals (e.g., UWB radar signals) and receive one or more echoes. Such radar circuitry 230 is configured to aid in generating radar data (e.g., time of flight, angle of transmission, angle of reception) for determining motion data of, for example, a user, virtual reality device 116-3, input device 120, or any combination thereof. To this end, radar circuitry 230 is configured to first generate one or more radar signals having one or more predetermined frequencies (e.g., between 3.1 GHz and 10.6 GHz). For example, radar circuitry 230 is configured to generate a series of pulses at one or more predetermined frequencies (e.g., between 3.1 GHz and 10.6 GHz). In embodiments, the radar circuitry 230 is configured to transmit one or more radar signals toward the virtual reality headset 116-3, the user, the input device 120, or any combination thereof. As an example, the radar circuitry 230 first determines whether the virtual reality headset 116-3, the user, the input device 120, or any combination thereof, is within range of the radar circuitry 230, for example, based on one or more communication protocols (e.g., by the virtual reality headset 116-3 reporting its location), one or more received echoes (e.g., indicating that the location of the virtual reality headset 116-3 is within range), one or more motion sensors (e.g., optical sensors, ultrasonic sensors, laser-based sensors), one or more user inputs, or any combination thereof. The radar circuitry 230 then transmits one or more radar signals (e.g., toward the virtual reality system 116-3) and receives one or more echoes (e.g., radar signals that have bounced off the virtual reality system 116-3, the user, the one or more input devices 120, or any combination thereof).Based on the echoes, the radar circuitry 230 determines radar data indicative of the time of flight of one or more radar and echo signals, the transmission angle of one or more radar signals, the reception angle of one or more echoes, or any combination thereof. Although the exemplary embodiment presented in FIG. 2 presents three ecosystem devices (228-1, 228-2, 228-N) each having a respective radar circuitry (230-1, 230-2, 230-N), in other embodiments, any number of ecosystem devices 228 in the smart ecosystem 232 may have a respective radar circuitry 230.

[0031] According to embodiments, the radar circuitry 230 is configured to transmit radar data indicative of the radar signals and echoes, radar data, or both, to a processing device in the client system 200. For example, the radar circuitry 230 transmits the radar data to an ecosystem device 228 in a smart ecosystem 232 that includes one or more processors. In response to receiving the radar data indicative of the radar signals and echoes, radar data, or both, the processing device is configured to determine motion data (position, movement, or both of a user, a virtual reality headset 116-3, one or more input devices 120). For example, the processing device compares the received radar data to data in one or more tables, arrays, vectors, libraries, or any combination thereof to determine the position and movement of the user and one or more input devices 120 (e.g., an input device held by a user). Furthermore, using such motion data, the processing device then determines one or more game inputs, for example, by comparing the motion data to data in one or more tables, arrays, vectors, libraries, or any combination thereof. In embodiments, the processing device then transmits the motion data, the game input, or both to the server 102 executing the game application associated with the displayed game stream 108 to modify the game application.

[0032] 3, an exemplary process 300 for implementing a radar for cloud-based gaming input is presented. The exemplary process 300 first includes a virtual reality headset 116-3 (e.g., a wearable display) receiving, over a network 110, a game stream 108 representing a gaming application executed by a server 102. In embodiments, the virtual reality display 116-3 includes or is otherwise connected to a display 334 including one or more displays (e.g., a liquid crystal display (LCD) screen, a light emitting diode (LED) screen, an organic light emitting diode (OLED) screen, an active matrix organic light emitting diode (AMOLED) screen, or any combination thereof) configured to display the game stream 108. As an example, in some embodiments, a client system (e.g., client system 112, 200) including the virtual reality headset 116-3 includes one or more decoders (e.g., decoder 114) configured to decode the game stream 108 received from the network 110 and produce a decoded game stream displayed by the display 334. To control the gaming application represented by the game stream 108 displayed on the display 334, the virtual reality headset 116-3 includes one or more sensors 336 configured to generate sensor data 305 indicative of a position of a user (e.g., a user wearing the virtual reality headset 116-3), a position of the virtual reality headset 116-3, a position of one or more input devices 120 (e.g., an input device held by a user), a movement (e.g., direction, speed, acceleration) of the user, a movement of the virtual reality headset, a movement of the one or more input devices, or any combination thereof. Such sensors 336 may include, for example, one or more optical sensors (e.g., photoelectric sensors, laser-based sensors, infrared sensors), ultrasonic sensors, Hall effect sensors, or any combination thereof.After generating the sensor data 305, the virtual reality headset 116-3 transmits the sensor data 305 to one or more processing devices 340 included within one or more client systems (e.g., client systems 112, 200). The processing devices 340 may include, for example, one or more processors each included within or otherwise connected to a respective client device 116 (e.g., computer, smartphone, ecosystem device, virtual reality headset) of the client system (e.g., client system 112, 200).

[0033] In addition, to facilitate control of a gaming application represented by the game stream 108 displayed on the display 334, the example process 300 includes one or more input devices 120 (e.g., input devices 120 held by a user, virtual reality controllers) receiving one or more interactions (e.g., gestures, button clicks, keystrokes, drags, rotations, movements, voice commands, noises) from a user (e.g., a user of a virtual reality headset 116-3). In response to receiving the one or more interactions, the input devices 120 transmit input data 310 representing the interactions to one or more processing devices 340. Additionally, to facilitate control of a gaming application represented by the game stream 108, the example process 300 includes an ecosystem device 228 having radar circuitry 230 and configured to generate and transmit radar data 315 to the processing device 340. To this end, in response to the virtual reality headset 116-3 receiving the game stream 108, the radar circuitry 230 of the ecosystem device 228 is configured to generate and transmit one or more radar signals in the direction of the virtual reality headset 116-3. Such radar signals then bounce off the virtual reality headset 116-3, the user, the input device 120 (e.g., the input device 120 held by the user), or any combination thereof, producing one or more echoes. In response to receiving the one or more echoes, the radar circuitry 230 determines radar data 315 based on the received echoes. The radar data 315 includes, for example, data indicative of a time of flight, an angle of transmission, an angle of reception, or any combination thereof, for one or more transmitted radar signals, received echoes, or both. For example, in response to receiving an echo, the radar circuitry 230 performs one or more operations based on the echoes (e.g., an operation based on the power of the echo, the angle of the echo, the time the echo was received) to determine the radar data 315.After determining the radar data 315, the ecosystem device 228 transmits the radar data 315 to the processing device 340.

[0034] In response to receiving the sensor data 305, the input data 310, the radar data 315, or any combination thereof, the processing device 340 is configured to determine one or more game inputs 320. Such game inputs include, for example, signals that can be interpreted by a gaming application to fulfill one or more commands. According to embodiments, to generate the game inputs 320, the processing device 340 is configured to compare the sensor data 305, the input data 310, the radar data 315, or any combination thereof, with data stored in one or more tables, arrays, vectors, libraries, or any combination thereof. As an example, the processing device 340 is configured to compare one or more interactions indicated in the input data 310 with data stored in one or more tables, arrays, vectors, libraries, or any combination thereof to determine one or more commands for the gaming application. The processing device 340 then generates the one or more game inputs 320 based on the determined commands.

[0035] Additionally, to generate game input 320, the processing device is configured to determine motion data for the virtual reality headset 116-3, the user, the one or more input devices 120, or any combination thereof based on the sensor data 305, the input data 310, the radar data 315, or any combination thereof. Such motion data may include, for example, data indicative of the position, movement (e.g., direction, velocity, acceleration), or both, of the user, the virtual reality headset 116-3, the one or more input devices 120, or any combination thereof. For example, based on one or more times of flight, transmission angles, receive angles, or any combination thereof indicated in the radar data 315, the processing device 340 is configured to determine a position and movement of the user based on a comparison of the times of flight, transmission angles, and receive angles indicated in the radar data 315 with one or more previously determined positions of the user, previously determined movements of the user, previously received times of flight, previously received transmission angles, previously received receive angles, or any combination thereof (e.g., as indicated in previously received radar data from the ecosystem device 228, as indicated in previously received sensor data 305, previously determined by the processing device 340, or any combination thereof). After determining motion data for the virtual reality headset 116-3, the user, the one or more input devices 120, or any combination thereof, the processing device 340 then generates one or more game inputs 320 based on the motion data. For example, the processing device 340 compares the user's movements indicated in the motion data to data in one or more tables, arrays, vectors, libraries, or any combination thereof to determine one or more commands for a gaming application. The processing device 340 then generates one or more game inputs 320 based on the determined commands. In this manner, the radar data 315 generated by the ecosystem device 228 is configured to generate game inputs 320 for a gaming application.In this manner, the radar data 315 is used to control the gaming application represented by the game stream 108 displayed on the virtual reality headset 116-3.

[0036] 4, an exemplary processing device 400 for implementing radar for cloud-based gaming input is presented. In embodiments, the exemplary processing device 400 includes one or more processors each included within or otherwise connected to a respective client device 116 (e.g., computer, smartphone, ecosystem device, virtual reality headset) of a client system (e.g., client system 112, 200) similar to or the same as processing device 340. According to embodiments, the exemplary processing device 400 includes controller circuitry 442 including hardware-based circuitry, software-based circuitry, or both configured to generate motion data 446 based on sensor data 305 (e.g., data indicative of a user's position, a virtual reality headset (e.g., wearable display) 116-3's position, one or more input devices' position, user's movement, virtual reality headset's movement, one or more input devices' movement), radar data 315 (e.g., data indicative of a time of flight, transmission angle, reception angle, or any combination thereof, of one or more transmitted radar signals, received echoes, or both), or both. Such motion data 446 may include, for example, data indicative of the position, movement (e.g., direction, speed, acceleration), or both, of a user, a virtual reality headset 116-3, one or more input devices 120, or any combination thereof. In some embodiments, to generate the motion data 446, the motion circuitry 444 is configured to determine the position of the virtual reality headset 116-3, the user, the one or more input devices, or any combination thereof relative to one or more ecosystem devices 228 based on the radar data 315.For example, in response to receiving radar data 315 from one or more respective ecosystem devices 228, the motion circuitry 444 is configured to determine a position of the virtual reality headset 116-3, the user, the one or more input devices, or any combination thereof, based on the time of flight, angle of transmission, angle of reception, or any combination thereof indicated in the received radar data 315 and based on the location of the one or more respective ecosystem devices 228 (e.g., the ecosystem device 228 that generated the radar data 315). That is, based on the location of the ecosystem device 228 and the radar data 315 generated by the ecosystem device 228 (time of flight, angle of transmission, angle of reception), the motion circuitry 444 is configured to determine the position of the virtual reality headset 116-3, the user, the one or more input devices, or any combination thereof, relative to the location of the ecosystem device.

[0037] In some embodiments, to generate the motion data 446, in some embodiments, the motion circuitry 444 is configured to compare the sensor data 305, the radar data 315, or both, to previously received sensor data (e.g., data indicative of previous positions and movements of the user, the virtual reality headset 116-3, and the input device 120), previously received radar data (e.g., data indicative of previous times of flight, angles of reception, and angles of transmission), previously determined motion data, or any combination thereof. For example, the motion circuitry 444 is configured to compare one or more movements of the virtual reality headset 116-3 indicated by the sensor data 305 with the previously determined motion data to determine a change in the position and movement of the virtual reality headset 116-3, a current position and movement of the virtual reality headset 116-3, or both. As another example, the motion circuitry 444 is configured to compare one or more times of flight indicated in the radar data 315 with the previously determined motion data to determine a change in the position of the user, a current position of the user, or both. After determining the motion data 446 , the motion circuitry 444 transmits the motion data 446 to the controller circuitry 442 .

[0038] The controller circuitry 442 includes hardware-based circuitry, software-based circuitry, or both configured to determine one or more game inputs 320 (e.g., signals interpretable as one or more commands by a gaming application) based on the sensor data 305, the input data 310 (e.g., data representing interactions with the input device 120), the movement data 446, or any combination thereof. For example, the controller circuitry 442 is configured to determine the game inputs 320 based on the input data 310 by comparing one or more interactions indicated in the input data 310 to data in one or more tables, arrays, vectors, libraries, or any combination thereof, determining one or more commands, and generating one or more game inputs 320 based on the commands. As another example, the controller circuitry 442 is configured to compare one or more angles, movements, or both indicated in the sensor data 305 to one or more tables, arrays, vectors, libraries, or any combination thereof, determining one or more commands, and generating one or more game inputs 320 based on the commands. As yet another example, the controller circuitry 442 is configured to compare one or more positions, movements, or both of the virtual reality headset 116-3, the user, the one or more input devices 120, or any combination thereof, indicated in the motion data 446 to one or more tables, arrays, vectors, libraries, or any combination thereof, determine one or more commands, and generate one or more game inputs 320 based on the commands.

[0039] 5, an exemplary method 500 for implementing a radar for cloud-based gaming input is presented. At step 505 of the exemplary method 500, a client system (e.g., client system 112, 200) including a virtual reality headset (e.g., virtual reality headset 116-3) and one or more input devices (e.g., input device 120) receives a game stream (e.g., game stream 108). The game stream represents a gaming application running on a server (e.g., server 102) connected to the client system by a network 110. In response to receiving the game stream, the client system decodes the game stream to produce a decoded game stream that is provided to a virtual reality headset (e.g., a wearable display). The virtual reality headset then displays the decoded game stream on a display (e.g., display 334). At step 510, a client device (e.g., client device 116) in the client system generates sensor data (e.g., sensor data 305), input data (e.g., input data 310), and radar data (e.g., radar data 315) for controlling a gaming application represented by the decoded game stream displayed on the virtual reality headset. To this end, the virtual reality headset of the client system includes one or more sensors (e.g., sensor 336) configured to generate sensor data indicative of a user's position, a virtual reality headset 116-3's position, one or more input devices' position, a user's movement, a virtual reality headset's movement, and one or more input devices' movement. For example, the virtual reality headset's sensors are configured to generate sensor data indicative of a movement and an angle of the virtual reality headset. Additionally, the input device of the client system is configured to receive one or more interactions (e.g., gestures, button clicks, keystrokes, drags, rotations, movements, voice commands, noises) and generate input data representative of the received interactions.

[0040] Still referring to step 510, the client system also includes a smart ecosystem (e.g., smart ecosystem 232) representing a group of devices (e.g., ecosystem devices 228) each having respective capabilities and networked together using one or more communication protocols (e.g., ZigBee, Bluetooth, Bluetooth LE, Wi-Fi) to provide their respective capabilities to one or more users. In embodiments, the virtual reality headset and input device of the client system are included within or otherwise communicatively coupled to the smart ecosystem. According to embodiments, the smart ecosystem of the client system includes one or more ecosystem devices configured to generate one or more radar signals. Such radar signals include, for example, one or more pulses having one or more predetermined frequencies (e.g., between 3.1 GHz and 10.6 GHz). The radar signals are configured to bounce off the virtual reality headset, the user, the one or more input devices, or any combination thereof, to produce one or more echoes (e.g., signals representative of the bounced radar signals). In response to receiving the one or more echoes, the ecosystem device is configured to generate radar data indicative of a time of flight, an angle of transmission, an angle of reception, or any combination thereof, of the one or more radar signals, echoes, or both. In embodiments, after generating the sensor data, input data, or radar data, respectively, the virtual reality headset, the input device, and the ecosystem device are configured to transmit the generated sensor data, input data, and radar data to a processing device (e.g., processing device 340, 400) of the client system. Such a processing device may include, for example, one or more processors each included within or otherwise connected to a respective client device 116 (e.g., computer, smartphone, ecosystem device, virtual reality headset) of the client system (e.g., client system 112, 200).

[0041] In response to receiving the radar data from one or more ecosystem devices at step 515, the processing device is configured to determine motion data (e.g., motion data 446) for the virtual reality headset, the user, the one or more input devices, or any combination thereof. Such motion data includes, for example, data indicative of a position, movement (e.g., direction, speed, acceleration), or both, of the user, the virtual reality headset, the one or more input devices, or any combination thereof. To generate the motion data, in embodiments, the processing device compares the received radar data to previously received radar data (e.g., data indicative of a previous time of flight, angle of reception, and angle of transmission), previously determined motion data, or both. For example, the processing device compares one or more times of flight indicated in the received radar data to previously determined motion data to determine a change in the user's position, the user's current position, or both. At step 520, the processing device determines one or more game inputs (e.g., signals that can be interpreted as one or more commands by a gaming application) based on the received sensor data, the received input data, the determined motion data, or any combination thereof. For example, the processing device compares the received sensor data, the received input data, the determined motion data, or any combination thereof, to data in one or more tables, arrays, vectors, libraries, or any combination thereof, and determines one or more commands. As an example, the processing device compares one or more positions and movements of the virtual reality headset indicated in the determined motion data to data in one or more tables, arrays, vectors, libraries, or any combination thereof, and determines one or more commands. Based on the determined commands, the processing device determines game inputs.

[0042] At step 525, the processing device sends the determined game input to a server executing a gaming application associated with the game stream to be displayed on the virtual reality headset. In response to receiving the game input, the server provides the game input to the gaming application and modifies the gaming application based on commands indicated in the game input. Based on the modified gaming application, the server generates a modified game stream and sends the game stream to the client device. At step 530, the client system receives the modified game stream, decodes the modified game stream, and provides the decoded modified game stream to the virtual reality headset for display.

[0043] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a flash memory, a cache, a solid-state storage device such as a random access memory (RAM), or one or more other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be in the form of source code, assembly language code, object code, or another form of instructions that can be interpreted or otherwise executed by one or more processors.

[0044] A computer-readable storage medium may include any storage medium, or combination of storage media, that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM), or cache), non-volatile memory (e.g., read-only memory (ROM), or flash memory), or microelectromechanical system (MEMS)-based storage media. A computer-readable storage medium may be incorporated within a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., magnetic hard drives), removably attached to a computing system (e.g., optical disks, or Universal Serial Bus (USB)-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).

[0045] It should be noted that not all of the above-mentioned activities or elements are required in the general description, that certain activities or portions of devices may not be required, and that one or more additional activities may be performed or one or more additional elements may be included in addition to those described. It should also be noted that the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various changes and modifications may be made without departing from the scope of the present disclosure as set forth in the appended claims. Thus, the specification and figures should be regarded in an illustrative and not a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0046] Benefits, or other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and any features that may cause or make any benefit, advantage, or solution more pronounced should not be construed as key, necessary, or essential features of any or all claims. Moreover, the particular embodiments disclosed herein are merely illustrative, as the subject matter of the present disclosure may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design shown herein, other than as described in the appended claims. It is therefore apparent that the particular embodiments disclosed above may be altered or modified, and all such variations are believed to be within the scope of the subject matter of the present disclosure. Accordingly, the protection sought herein is as set forth in the appended claims.

Claims

1. 1. A method for cloud based gaming, comprising: Transmitting a radar signal in response to the wearable display receiving the game stream; determining movement data of a user of the wearable display from the radar signals; and generating one or more game inputs based on the movement data of the user of the wearable display; The method includes:

2. modifying the game stream based on the one or more game inputs; transmitting the modified game stream to the wearable display; The method of claim 1 further comprising:

3. generating, at the wearable display, sensor data associated with the user; The method of claim 1 or 2, wherein the one or more game inputs are further determined based on the sensor data associated with the user.

4. receiving, at the input device, one or more interactions; The method of any one of claims 1 to 3, wherein the one or more game inputs are further determined based on input data representative of the one or more interactions.

5. The method of any one of claims 1 to 4, further comprising transmitting the one or more game inputs to a cloud-based server associated with the game stream.

6. The method of any one of claims 1 to 5, wherein the movement data indicates a position of the user of the wearable display.

7. The method of any preceding claim, wherein the movement data indicates a position of the wearable display.

8. The method of any one of claims 1 to 7, wherein the wearable display is connected to a smart ecosystem including an ecosystem device configured to transmit the radar signal.

9. The method of claim 8 , wherein the ecosystem device includes at least one of a smart hub, a display, or a sensor.

10. The method of any one of claims 1 to 9, wherein the radar signal comprises an ultra-wideband radar signal.

11. 1. A method for cloud based gaming, comprising: In response to initiating a client gaming session, receiving from a wearable display sensor data associated with a user of the wearable display and from an ecosystem device radar data associated with the user of the wearable display; determining one or more game inputs based on the sensor data and the radar data associated with the user of the wearable display; receiving a modified game stream based on the one or more game inputs in response to transmitting the one or more game inputs to a cloud-based server associated with the client gaming session; and The method includes:

12. generating a game stream at the cloud-based server associated with the client gaming session; modifying the game stream based on the one or more game inputs; The method of claim 11 further comprising:

13. The method of claim 11 or 12, wherein the wearable display and the ecosystem device are in a smart ecosystem.

14. The method of any one of claims 11 to 13, wherein the ecosystem device includes radar circuitry configured to transmit an ultra-wideband radar signal.

15. receiving input data from an input device representing one or more interactions with the input device; The method of any one of claims 11 to 14, wherein the one or more game inputs are further determined based on the input data.

16. Transmitting a radar signal; and In response to receiving echoes of the radar signal, determining the radar data associated with the user of the wearable display based on the echoes of the radar signal; The method according to any one of claims 11 to 15, further comprising:

17. The method of any one of claims 11 to 16, wherein the ecosystem device includes at least one of a smart hub, a display, or a sensor.

18. determining motion data associated with the user of the wearable display based on the radar data; The method of any one of claims 11 to 17, wherein the one or more game inputs are further determined based on the movement data.

19. The method of claim 18 , wherein the motion data indicates a position of the user of the wearable display.

20. The method of claim 18 or 19, wherein the movement data indicates a position of the wearable display.

21. one or more cloud-based gaming servers communicatively coupled to one or more client systems, each client system comprising: a wearable display including one or more processors; and A cloud-based gaming system including one or more ecosystem devices, at least one of the ecosystem devices including one or more processors and a memory coupled to the one or more processors and storing executable instructions configured to operate the one or more processors to perform a method according to any one of claims 1 to 20.

22. one or more cloud-based gaming servers communicatively coupled to one or more smart ecosystems, each smart ecosystem comprising: a wearable display including one or more processors; and A cloud-based gaming system including one or more ecosystem devices, at least one of the ecosystem devices including one or more processors and a memory coupled to the one or more processors and storing executable instructions configured to operate the one or more processors to perform a method according to any one of claims 1 to 20.

Images