System and method for performing action in response to change in status of input device

Abstract

To provide systems and methods for performing an action in response to a change in status of an input device.SOLUTION: A method includes: generating a media item for output at a computing device; identifying a threshold based on a status of an input device associated with the computing device; identifying a change in status of the input device, the change exceeding the threshold; receiving an indication of the change in status of the input device; and, in response to receiving the indication, performing an action based on the media item. The media item is a video game and performing the action may further comprise pausing the video game and generating an advertisement for output. Generating an advertisement for output may further comprise altering an element of the video game based on the advertisement.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure is directed to systems and methods for performing actions in response to changes in the status of an input device. In particular, provided herein are systems and methods for enabling actions to be performed in response to changes in the status of an input device based on identified thresholds. [Background technology]

[0002] The proliferation of computing devices such as next generation gaming consoles and smart televisions has led to an increase in available computing power and therefore an increase in the range of actions and tasks that these devices can perform. However, not all computing devices efficiently utilize idle time and / or resources when a user is absent and / or present. For example, a gaming console may only launch updates at night, rather than during downtime between gaming sessions. In another example, a smart television may display advertisements when no user is present in the room. Part of the problem is that many computing devices are unable to determine whether a user is engaged with the output of the computing device (e.g., viewing a content item) or whether the user is about to stop engaging with the output of the computing device (e.g., leaving the room). Furthermore, many computing devices are unable to determine whether a user is about to perform an action (e.g., fast forwarding an advertisement). These problems are typically difficult for computing devices to address because the computing device has incomplete information about how a user is engaged or about to engage with the output of the computing device.

[0003] To overcome these problems, systems and methods are provided herein for performing actions in response to changes in the status of an input device. Summary of the Invention [Means for solving the problem]

[0004] Systems and methods are described herein for enabling actions to be performed in response to changes in a status of an input device. A media item is generated for output at a computing device and a threshold is identified. The threshold is based on a status of the input device over a period of time, the input device being associated with the computing device. A change in the status of the input device that exceeds the threshold is identified and an indication of the change in the status of the input device is received. In response to receiving the indication, an action based on the media item is performed.

[0005] In an exemplary system, a user turns on a gaming console and loads a video game. The video game is generated for output at the gaming console and displayed via a display connected to the gaming console. The user interacts with the gaming console via a wireless controller. When the user is using the gaming console, the controller moves around. The movement may vary depending on the action on the screen, but as long as the controller is held by the user, the controller will move to various degrees. Over a period of time, a threshold may be identified based on the movement of the controller. The threshold may relate to controller position change, controller orientation change, controller travel distance, duration of controller movement or lack thereof, or rate of movement for the controller over time. Regardless of the exact nature or value of the threshold, the threshold may be utilized by the system to determine that the controller has changed state. In some instances, the state or status of the controller represents the state of the movement (or lack thereof) of the controller. The state may be considered to indicate whether the controller is in use. For example, when a user puts the controller down to eat a snack, the controller may be considered to be stationary. The stationary state suggests that the console or system corresponding to the controller is not in use. In this example, a change in the status of the controller from moving to stationary may be determined to exceed a threshold, and an indication of the change in the status of the controller or input device is received. In some examples, this may include receiving an indication from the controller itself. In other examples, this may include one component of the gaming console or system (e.g., a hardware and / or software component) receiving an indication from another component of the gaming console (e.g., an infrared, lidar, or image sensor of the console). In response to receiving the indication of the change in status, an action is performed based on the media item.For example, a video game may be paused and an advertisement may be generated for output at the gaming console.

[0006] The media item may be a video game, and performing the action may further include pausing the video game and generating an advertisement for output. Generating the advertisement for output may further include modifying an element of the video game based on the advertisement. The change in status of the input device may be a first change in status, and performing the action may further include identifying a second change in status of the input device and generating a content item for output based on the video game gameplay between the first change in status and the second change in status. Performing the action may further include identifying a second change in status of the input device and modifying a status of a character of the video game between the first change in status and the second change in status.

[0007] Identifying a change in the status of the input device may further include identifying movement of the input device, or may further include identifying an absence of movement of the input device for a threshold period of time.

[0008] Performing the action may further include generating a plurality of advertisement summaries and generating the plurality of advertisement summaries for output. Performing the action may further include identifying that an advertisement is being generated for output and generating a user interface element for output, the user interface element configured to allow input for the advertisement to be received. A user profile may be associated with the computing device, and performing the action may further include generating the advertisement for output based on the user profile. A time period for performing the action may be identified based on the change in status.

[0009] In some cases, the input device is a remote control device and the corresponding system that receives input via the input device is a media platform or system that delivers video (e.g., an over-the-top or OTT box, e.g., a set-top box that provides cable service, or a television, such as a smart television, that includes an operating system configured to enable video delivery and display). The present invention provides, for example, the following items. (Item 1) 1. A method comprising: generating, at a computing device, a media item for output; identifying a threshold value, the threshold value being based on a status of an input device over a period of time, the input device being associated with the computing device; identifying a change in a status of the input device that exceeds the threshold; receiving an indication of a change in status of the input device; performing an action based on the media item in response to receiving the indication; A method comprising: (Item 2) The method of any preceding claim, wherein the media item is a video game and performing the action further comprises pausing the video game and generating an advertisement for output. (Item 3) 20. The method of claim 19, wherein generating an advertisement for output further comprises altering an element of the video game based on the advertisement. (Item 4) the media item is a video game; the change in status of the input device is a first change in status, Taking the above actions further: identifying a second change in status of the input device; and generating a content item for output based on gameplay of the video game between the first change in status and the second change in status; and The method according to any one of the above items, comprising: (Item 5) the media item is a video game; the change in status of the input device is a first change in status, Taking the above actions further: identifying a second change in status of the input device; and modifying a status of the video game character between the first change in status and the second change in status; The method according to any one of the above items, comprising: (Item 6) 13. The method of claim 1, wherein identifying a change in the status of the input device further comprises identifying a movement of the input device. (Item 7) 13. The method of claim 1, wherein identifying a change in the status of the input device further comprises identifying an absence of movement of the input device for a threshold period of time. (Item 8) Taking the above actions further: generating a summary of a plurality of advertisements; generating a summary of the plurality of advertisements for output; and The method according to any one of the above items, comprising: (Item 9) Taking the above actions further: identifying an advertisement being generated for output; generating a user interface element for output, the user interface element configured to allow input for the advertisement to be received; The method according to any one of the above items, comprising: (Item 10) A user profile is associated with the computing device; performing the action further includes generating an advertisement for output based on the user profile. The method according to any one of the above items. (Item 11) 13. The method of claim 1, further comprising identifying a time period for performing the action based on the change in status. (Item 12) 1. A system comprising: A communication port; A memory for storing instructions; control circuitry communicatively coupled to the memory and to the communication port, the control circuitry comprising: generating, at a computing device, a media item for output; identifying a threshold value, the threshold value being based on a status of an input device over a period of time, the input device being associated with the computing device; identifying a change in a status of the input device that exceeds the threshold; receiving an indication of a change in status of the input device; performing an action based on the media item in response to receiving the indication; and control circuitry configured to execute instructions to A system comprising: (Item 13) The system of any one of the preceding claims, wherein the media item is a video game and the control circuitry configured to perform the action is further configured to pause the video game and generate an advertisement for output. (Item 14) 20. The system of claim 1, wherein the control circuitry configured to generate an advertisement for output is further configured to modify an element of the video game based on the advertisement. (Item 15) the media item is a video game; the change in status of the input device is a first change in status, The control circuitry configured to perform the actions further comprises: identifying a second change in status of the input device; and generating a content item for output based on gameplay of the video game between the first change in status and the second change in status; and 2. The system of claim 1, configured to: (Item 16) the media item is a video game; the change in status of the input device is a first change in status, The control circuitry configured to perform the actions further comprises: identifying a second change in status of the input device; and modifying a status of the video game character between the first change in status and the second change in status; 2. The system of claim 1, configured to: (Item 17) 20. The system of claim 19, wherein the control circuitry configured to identify a change in status of the input device is further configured to identify movement of the input device. (Item 18) 20. The system of claim 19, wherein the control circuitry configured to identify a change in status of the input device is further configured to identify an absence of movement of the input device for a threshold period of time. (Item 19) The control circuitry configured to perform the actions further comprises: generating a summary of a plurality of advertisements; generating a summary of the plurality of advertisements for output; and 2. The system of claim 1, configured to: (Item 20) The control circuitry configured to perform the actions further comprises: identifying an advertisement being generated for output; generating a user interface element for output, the user interface element configured to allow input for the advertisement to be received; 2. The system of claim 1, configured to: (Item 21) A user profile is associated with the computing device; the control circuitry configured to perform the action is further configured to generate an advertisement for output based on the user profile. A system according to any one of the preceding items. (Item 22) 20. The system of claim 1, wherein the control circuitry is further configured to identify a time period for performing the action based on the change in status. (Item 23) A non-transient computer readable medium having non-transient computer readable instructions encoded thereon that, when executed by control circuitry, cause the control circuitry to: generating, at a computing device, a media item for output; identifying a threshold value, the threshold value being based on a status of an input device over a period of time, the input device being associated with the computing device; identifying a change in a status of the input device that exceeds the threshold; receiving an indication of a change in status of the input device; performing an action based on the media item in response to receiving the indication; A non-transitory computer readable medium for causing (Item 24) The non-transitory computer-readable medium of any one of the preceding claims, wherein the media item is a video game, and execution of the instructions to perform the action further causes the control circuitry to pause the video game and generate an advertisement for output. (Item 25) 20. The non-transitory computer-readable medium of claim 1, wherein execution of the instructions for generating an advertisement for output further causes the control circuitry to modify an element of the video game based on the advertisement. (Item 26) the media item is a video game; the change in status of the input device is a first change in status, Execution of the instructions to perform the actions further includes causing the control circuitry to: identifying a second change in status of the input device; and generating a content item for output based on gameplay of the video game between the first change in status and the second change in status; and The non-transitory computer-readable medium of any one of the preceding items, (Item 27) the media item is a video game; the change in status of the input device is a first change in status, Execution of the instructions to perform the actions further includes causing the control circuitry to: identifying a second change in status of the input device; and modifying a status of the video game character between the first change in status and the second change in status; The non-transitory computer-readable medium of any one of the preceding items, (Item 28) 20. The non-transitory computer-readable medium of claim 1, wherein execution of the instructions to identify a change in status of the input device further causes the control circuitry to identify a movement of the input device. (Item 29) 20. The non-transitory computer-readable medium of claim 1, wherein execution of the instructions to identify a change in status of the input device further causes the control circuitry to identify an absence of movement of the input device for a threshold period of time. (Item 30) Execution of the instructions to perform the actions further includes causing the control circuitry to: generating a summary of a plurality of advertisements; generating a summary of the plurality of advertisements for output; and The non-transitory computer-readable medium of any one of the preceding items, (Item 31) Execution of the instructions to perform the actions further includes causing the control circuitry to: identifying an advertisement being generated for output; generating a user interface element for output, the user interface element configured to allow input for the advertisement to be received; The non-transitory computer-readable medium of any one of the preceding items, (Item 32) A user profile is associated with the computing device; 20. The non-transitory computer-readable medium of claim 1, wherein execution of the instructions to perform the actions further causes the control circuitry to generate an advertisement for output based on the user profile. (Item 33) 20. The non-transitory computer-readable medium of claim 1, wherein execution of the instructions further causes the control circuitry to identify a time period for performing the action based on the change in status. (Summary) A system and method are provided for performing an action in response to a change in a status of an input device, wherein a media item is generated for output at a computing device, a change in the status of an input device associated with the computing device is identified, an indication of the change in the status of the input device is received, and in response to receiving the indication, an action is performed based on the media item. [Brief description of the drawings]

[0010] The present disclosure will be described in detail with reference to the following figures, according to one or more various embodiments. The drawings are provided for illustrative purposes only and merely depict exemplary or exemplary embodiments. These drawings are provided to facilitate understanding of the concepts disclosed herein and are not to be considered as limiting the scope, scope, or applicability of these concepts. Please note that for clarity and ease of illustration, these drawings are not necessarily made to scale.

[0011] The above and other objects and advantages of the present disclosure will become apparent upon consideration of the following detailed description taken in conjunction with the accompanying drawings.

[0012] [Figure 1] FIG. 1 illustrates an example environment in which actions are performed in response to changes in the status of an input device, according to some embodiments of the disclosure.

[0013] [Diagram 2] FIG. 2 illustrates another example environment in which actions are performed in response to changes in the status of an input device, according to some embodiments of the present disclosure.

[0014] [Diagram 3] FIG. 3 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure.

[0015] [Figure 4] FIG. 4 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure.

[0016] [Diagram 5] FIG. 5 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure.

[0017] [Figure 6] FIG. 6 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure.

[0018] [Figure 7] FIG. 7 illustrates an example environment in which a change in the status of an input device is identified, according to some embodiments of the present disclosure.

[0019] [Figure 8] FIG. 8 shows a block diagram representing components of a computing device and data flow therebetween for enabling actions to be performed in response to changes in the status of an input device, according to some embodiments of the present disclosure.

[0020] [Figure 9] FIG. 9 shows a flowchart of illustrative steps involved in enabling actions to be performed in response to changes in the status of an input device, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Detailed Description Systems and methods are described herein for enabling actions to be performed in response to changes in the status of an input device. A media item includes audio, video, text, video games, and / or any other media content. A content item is a type of media item that typically includes an audiovisual component. A media item may be a single media item. In another embodiment, it may be a series (or season) of episodes of a media content item. Audio includes audio-only content, such as podcasts. Video includes audiovisual content, such as movies and / or television programs. Text includes text-only content, such as event descriptions. One embodiment of a suitable media content item is one that complies with the MPEG DASH standard. OTT streaming and / or video-on-demand (VOD) services (or platforms) may be accessed via websites and / or apps running on a computing device, and the device may receive any type of content item, including live and / or on-demand content items. A media item may be streamed, for example, to a physical computing device. In another embodiment, a media item may be streamed, for example, to a virtual computing device, for example, in an augmented environment, a virtual environment, and / or a metaverse.

[0022] The input device includes any wired and / or wireless controller for a computing device. For example, this may include a wireless controller for a gaming console and / or a remote control for a television. The input device may comprise an inertial measurement unit (IMU) or any other means for detecting the movement of the input device. The input device may comprise an infrared component, a Bluetooth component, a Wi-Fi component, and / or a touch screen. The input device may be a keyboard, a mouse, and / or a microphone.

[0023] Identification of thresholds and / or status changes of an input device may be performed at the input device itself and / or at a server remote from the input device and / or local computing device based on data transmitted to a local computing device associated with the input device. In some examples, the input devices may have pre-generated profiles associated with them that are stored in the server. Upon identifying the input device, the local computing device may request the profile from the server and may determine threshold and / or status changes based on the profile. In some examples, these pre-generated profiles may be provided by the manufacturer of the input device. In another example, data may be collected from multiple identical or similar input devices and threshold and / or status changes may be identified in a server for the multiple input devices. In other examples, the input device may comprise two components that are physically separate but communicate with each other. In some examples, identifying threshold and / or status changes may be based on one or both of the two components.

[0024] The disclosed methods and systems may be implemented on one or more computing devices. As referred to herein, a computing device may be any device that includes a processor and a memory, such as a television, a smart television, a set-top box, an integrated receiver decoder (IRD) for handling satellite television, a digital storage device, a digital media receiver (DMR), a digital media adapter (DMA), a streaming media device, a DVD player, a DVD recorder, a connected DVD, a local media server, a BLU-RAY® player, a BLU-RAY® recorder, a personal computer (PC), a laptop computer, a tablet computer, a WebTV box, a personal computer television (PC / TV), a PC media server, a PC media center, a handheld computer, a landline phone, a personal digital assistant (PDA), a mobile phone, a portable video player, a portable music player, a portable gaming device, a smartphone, a smart watch, a smart speaker, an augmented reality device, a mixed reality device, a virtual reality device, a gaming console, or any other television, computing, or wireless device, and / or a combination thereof.

[0025] The methods and / or any instructions for implementing any of the embodiments discussed herein may be encoded on a computer-readable medium. A computer-readable medium includes any medium capable of storing data. A computer-readable medium may be transitory, including but not limited to a propagating electrical or electromagnetic signal, or may be non-transient, including but not limited to volatile and non-volatile computer memory or storage devices such as hard disks, floppy disks, USB drives, DVDs, CDs, media cards, registered memory, processor cache, random access memory (RAM), and the like.

[0026] FIG. 1 illustrates an exemplary environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the disclosure. The environment 100 includes a computing device, in this example a gaming console 102, a display 104, and a controller 106, in this example with an IMU 108. FIG. 1 depicts a first scenario 118 in which the controller 106 is in use and a second scenario 120 in which the controller 106 is out of use. The gaming console 102 generates gaming output that is displayed via the display 104. In some examples, the gaming console 102 is connected to a network, such as the Internet. Media items may be received via the Internet, including content items, such as games and movies and / or television programs streamed via the cloud. A user interacts with the gaming console 102 via the controller 106. In this embodiment, the controller 106 is a wireless controller that is connected to the gaming console via, for example, a Bluetooth connection, however, in other examples, the controller may be connected to the gaming console via wired means. A threshold is identified (110) based on the status of the controller 106 over a period of time. In a first scenario 118, a threshold is identified based on the movement of the controller 106 as the user interacts with the controller 106 as the user plays a game on the gaming console 102. In some embodiments, the threshold may be based on the average movement of the controller 106 over a period of time. In some embodiments, the period may be 5 seconds, 30 seconds, and / or 1 minute. In another embodiment, the threshold may be based on a number of measurements of movement in a particular direction in three-dimensional space. In some instances, the system may indicate a threshold that indicates a minimum distance the controller 106 must move to change state or status (e.g., from "active" or "moving" to "stationary" or vice versa).By exceeding the threshold, the movement of the controller 106 may indicate that the controller 106 is, for example, being picked up or put down. In some instances, the threshold may indicate a minimum period of time that the controller 106 must be relatively immobile or relatively moving before a status or state change is recognized. In such an embodiment, the system utilizes multiple thresholds. For example, the system may utilize (i) a first threshold related to the distance or movement of the controller that may be utilized to determine whether the controller is considered to be moving and (ii) a time-related threshold indicating the time of distance or movement that must be detected before a state or status transition can be determined. In an embodiment, the movement of the controller 106 may be considered to be moving only when a sufficient location change is detected (e.g., a location change of 2 inches or more may be required) or when a sufficient orientation change is detected (e.g., the controller may require a 15 degree orientation change before the controller 106 is considered to be moving). The orientation or position of the controller 106 may be detected via any suitable sensor of the controller 106 or corresponding system 102. For example, the controller 106 may include an IMU or GPS system to detect controller movement. The system 102 may include an IR sensor, a laser sensor, or an image sensor that may be utilized to track the movement of the controller 106. In some embodiments, the threshold indicates the number of detected movements for the controller 106 over a given period of time. In an embodiment, the controller 106 is a controller for a media platform or system that delivers video (e.g., an over-the-top or OTT box, e.g., a set-top box that provides cable service, or a television, such as a smart television, that includes an operating system configured to enable video distribution and display).

[0027] In any event, the threshold may be used to determine that a change in state or status of the controller 106 has occurred. For example, when the user puts the controller 106 down in the second scenario 120, a change in status is identified (112) based on the threshold. In this embodiment, the system attempts to identify that the controller 106 has changed from a state in which it is substantially moving (i.e., being used for gaming) to a state in which it is substantially stationary (i.e., it is being put down). The change in status may be identified in the controller 106 itself. In another embodiment, data may be transmitted from the controller 106 to the gaming console 102 and the change in status may be identified in the gaming console 102. The movement of the controller for one or both of the identifying steps 110, 112 may be identified via the IMU 108. In other embodiments, the controller 106 may include any sensor suitable for detecting a change in movement. In another embodiment, the wireless signal between the controller 106 and the gaming console 102 may be monitored, for example, to identify a change in signal strength. In some embodiments, changes in signal strength may be determined via a received signal strength indicator (RSSI), which may increase or decrease based on movements of the controller 106. In some embodiments, trained machine learning algorithms may be used to associate various controller 106 movements with changes in thresholds and / or status of the controller 106. The trained machine learning algorithms may reside on the gaming console 102 and / or at a server remote from the gaming console 102.

[0028] Upon identifying a change in status, an indication of the change in status is received (114). In some embodiments, this may include receiving an indication from the controller itself. In other embodiments, this may include one component (e.g., a hardware and / or software component) of the gaming console 102 receiving an indication from another component of the gaming console 102. Upon receiving the indication of the change in status, an action 116 to be performed is determined and subsequently performed. The action to be performed may be determined at the gaming console 102. In another embodiment, data may be transmitted over a network, such as the Internet, to a server, and the action 116 to be performed may be determined at the server. Upon determining the action 116 to be performed, the server may transmit the action to the gaming console 102 where the action is performed. The action 116 may comprise a single action. In another embodiment, the action 116 may comprise multiple actions, which may be performed sequentially and / or in parallel.

[0029] FIG. 2 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure. The environment 200 includes a computing device, in this example a smart television 202, a remote control device 204 in communication with the smart television 202, and a camera 206. FIG. 2 depicts a first scenario 216 in which the remote control device 204 is in an unused state and a second scenario 218 in which the remote control device 204 is in use. The smart television 202 generates content item outputs that are displayed via a display of the smart television 202. In some examples, the smart television 202 is connected to a network such as the Internet. Media items, including content items such as movies and / or television programs, may be received via the Internet. A user interacts with the smart television 202 via the remote control device 204. In this example, the remote control device 204 is a wireless controller that communicates with the smart television 202 via infrared. The movement or orientation of the remote control device 204 may be tracked in a manner similar to that described with reference to the controller 106 depicted in Figure 1. Additionally, one or more thresholds similar to or the same as those described with reference to Figure 1 may be utilized to determine whether a change in the status of the remote control device 204 has occurred.

[0030] A threshold is identified (208) based on the status of the controller 204 over a period of time. In a first scenario 216, a threshold is identified based on movement of the remote control device 204 while the user is viewing a content item on the smart television 202. In some embodiments, the threshold may be based on an average movement of the remote control device 204 over a period of time. In some embodiments, the period may be 5 seconds, 30 seconds, and / or 1 minute. Movement may be measured using any suitable speed measurement (e.g., mph), distance measurement (e.g., inches or cm), acceleration measurement (e.g., meters per second per second), position measurement (e.g., coordinate position), or orientation measurement (e.g., measured in degrees relative to any suitable plane). In some embodiments, the threshold may require a minimum of either 15 degree rotations observed over 5 seconds or 4 inches traveled over 10 seconds. In some instances, the threshold may be represented by one or more logical OR statements. For example, the motion threshold may be exceeded when any of the following are true: (exceeds threshold 1) OR (exceeds threshold 2) OR (exceeds threshold 3). In another embodiment, the threshold may be based on the number of measurements of movement in a particular direction in three-dimensional space. When the user lifts the remote control device 204 in the second scenario 218, a change in status is identified (210) based on the threshold. In this embodiment, the system attempts to identify that the remote control device 204 has changed from a state in which it is substantially stationary (i.e., not being used while a content item is being viewed) to a state in which it is moving (i.e., it is being lifted). In addition to the method of identifying a change in the status of the controller described in connection with FIG. 1, the camera 206 may be used to monitor the movement of the controller. Captures from the camera 206 may be analyzed by a trained machine learning algorithm to determine whether the controller is moving or stationary. In some embodiments, the analysis may be performed in the smart TV 202.In other embodiments, output from the camera 206 may be streamed from the smart TV 202 over a network, such as the Internet, to a remote server, and the trained machine learning algorithm may be launched on the server. An example of how the trained machine learning algorithm may be utilized to identify changes in thresholds and / or status of a controller, such as the remote control device 204, is described below in connection with FIG.

[0031] Upon identifying a change in status, an indication of the change in status is received (212). In some embodiments, this may include receiving an indication from the controller itself. In other embodiments, this may include one component (e.g., a hardware and / or software component) of the smart TV 202 receiving an indication from another component of the smart TV 202. Upon receiving the indication of the change in status, an action 214 to take is determined and subsequently taken. The action to take may be determined at the smart TV 202. In another embodiment, the data may be transmitted over a network, such as the Internet, to a server, and the action 214 to take may be determined at the server. Upon determining the action 214 to take, the server may transmit the action to the smart TV 202 where the action is taken. The action 214 may comprise a single action. In another embodiment, the action 214 may comprise multiple actions, which may be performed sequentially and / or in parallel.

[0032] FIG. 3 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure. In a manner similar to the environment 100 discussed in connection with FIG. 1, the environment 300 includes a gaming console 302, a display 304, and a controller 306, which in this example includes an IMU 308. FIG. 3 depicts a first scenario 322 in which the controller 306 is in use and a second scenario 324 in which the controller 306 is out of use. The gaming console 302 generates gaming outputs that are displayed via the display 304. A user interacts with the gaming console 302 via the controller 306. In this embodiment, the controller 306 is a wireless controller that is connected to the gaming console via, for example, a Bluetooth connection, however, in other examples, the controller may be connected to the gaming console via wired means. Movement or orientation of the controller 306 may be tracked in a manner similar to that described with reference to the controller 106 depicted in FIG. 1. Additionally, one or more thresholds similar to or the same as those described with reference to FIG. 1 may be utilized to determine whether a change in the status of the controller 306 has occurred.

[0033] A threshold is identified (310) based on the status of the controller 306 over a period of time. When the user puts down the controller 306 in a second scenario 324, a change in status is identified (312) based on the threshold. Upon identifying the change in status, an indication of the change in status is received (314). Upon receiving the indication of the change in status, an action to take is determined and subsequently taken. In this example, two actions 316a, 316b to take are depicted; however, in some embodiments, it is contemplated that only one of the actions 316a, 316b is taken on the gaming console 302. The first action 316a is to generate a full screen advertisement 316a for display. In some embodiments, a media item launching on the gaming console 302 is automatically paused before the full screen advertisement is displayed. In some embodiments, the full screen advertisement 316 may comprise a service message 318, such as a message indicating that the controller 306 should be lifted to resume the media item. In another embodiment, the service message may comprise statistics and / or data corresponding to a gaming session, such as the number of players remaining in a multiplayer game and / or the player who is currently winning. The advertisement may be based on the media item, in that the advertiser may indicate that the advertisement may be displayed when the media item is paused. The second action 316b is to alter an element of the media item for display. In some embodiments, the media item launching on the gaming console 302 is automatically paused before the element of the media item is altered. In this embodiment, a character in the video game is altered to drink the advertised beverage 320. For example, any element of the video game can be altered, including the character's image and / or equipment. In some embodiments, the character may be altered to include equipment that is not currently purchased but is available for purchase.In one embodiment, the character may be modified to include a hat, and a corresponding message, e.g., "This hat is currently on sale in the store," may be displayed. In some embodiments, a vehicle associated with the character may be modified to display an advertisement and / or logo for the product. In another embodiment, the character's clothing may be modified to reflect the color of the item being advertised. In some embodiments, the character's attributes may be modified based on the time the video game is paused. In some embodiments, the gaming console may alternate between the first action 316a and the second action 316b. For example, a full screen advertisement may be generated for display for, e.g., 30 seconds, and a video game character drinking a beverage 320 may be generated for display for the following 30 seconds. In some embodiments, the character drinking the beverage 320 may be stationary. In other embodiments, the character drinking the beverage 320 may, e.g., drink the beverage 320 in a loop. One or both of the first and second actions 316a, 316b may have one or more audio components associated therewith.

[0034] FIG. 4 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure. In a manner similar to the environments 100, 300 discussed in connection with FIGS. 1 and 3, the environment 400 includes a gaming console 402, a display 404, and a controller 406, which in this example includes an IMU 408. FIG. 4 depicts a first scenario 424 in which the controller 406 is in use and a second scenario 426 in which the controller 406 is out of use. The gaming console 402 generates gaming outputs that are displayed via the display 404. A user interacts with the gaming console 402 via the controller 406. In this embodiment, the controller 406 is a wireless controller that is connected to the gaming console via, for example, a Bluetooth connection, however, in other examples, the controller may be connected to the gaming console via wired means. The movement or orientation of the controller 406 may be tracked in a manner similar to that described with reference to the controller 106 depicted in Figure 1. Additionally, one or more thresholds similar to or the same as those described with reference to Figure 1 may be utilized to determine whether a change in the status of the controller 406 has occurred.

[0035] A threshold is identified (410) based on the status of the controller 406 over a period of time. When the user puts down the controller 406 in a second scenario 426, a change in status is identified (412) based on the threshold. Upon identifying the change in status, an indication of the change in status is received (414). Upon receiving the indication of the change in status, an action to take is determined and subsequently taken. In this example, two actions 416a, 416b to take are depicted, however, in some examples, it is contemplated that only one of the actions 416a, 416b is taken on the gaming console 402. The first action 416a includes pausing a video game running on the gaming console 402 and generating a resume screen on the display 404. The resume screen allows the user to select a first user interface element 418, which allows the user to resume playing the game in a typical manner. The resume screen also allows the user to select a second user interface element 420, which allows the user to watch a replay of any action the user missed while the game was paused. For example, if the media item is a multiplayer video game, the game is paused for the current user, but the game may continue in the background for other players. Selecting the replay user interface element 420 may cause a short replay to be generated and displayed on the display 404 before the game is resumed. In some embodiments, the replay may be a speeded-up playback of one of the other players' view of the game. For example, if the game was paused for a minute, the replay may be a 2x speed replay so that it fits in 30 seconds. In another embodiment, important plays may be identified and the replay may comprise a rendering of the important plays. The replay may be generated at the gaming console 402 and / or transmitted to the gaming console 402 from a remote server over a network, such as the Internet.In another embodiment, the replay may be generated from frames stored in a buffer of the gaming console 402. The replay may include a timer to indicate when live gameplay will resume. The second action 416b is to pause the video game and change the status of a character 424 (e.g., the user's character) of the video game. Changing the status of the character 424 may include making the character invisible and / or invincible while the video game is paused and / or for a period of time after the video game is resumed. In addition, a service message 422 may be generated for display, for example indicating that the controller 406 should be lifted to resume the game. In some embodiments, the actions 416a and 416b may be combined such that the status of the character 424 is changed and a replay is generated. One or both of the first and second actions 416a, 416b may have one or more audio components associated with it.

[0036] Any of the actions 316a, 316b, 416a, 416b may be combined in any suitable manner. Additionally, other actions may occur and be performed at the gaming console 302, 402. Other actions include, but are not limited to, preloading any advertisements and / or elements of the media item and performing software updates associated with the gaming console 402 and / or the media item (e.g., updating software and / or components associated with the video game). In some embodiments, the action or actions may be performed at the controller and / or a secondary computing device associated with the gaming console 302, 402. For example, the controller 306, 406 and / or a secondary computing device such as a smartwatch may include a display on which advertisements are displayed. In this embodiment, the advertisements may be transmitted from the gaming console 302, 402 via local wireless means. In other embodiments, the advertisements may be received from a server over a network, such as the Internet, in response to a communication sent from the gaming console 302, 402 to the server over the network. In another embodiment, the secondary computing device may be a smart speaker and the action may include playing audio on the smart speaker, e.g., advertising audio and / or a voice message indicating the status of the gaming console 302, 402, such as "gaming console turned on." In some embodiments, a user may provide audio input to the smart speaker to perform an additional action, such as turning off the gaming console 302, 402. In some embodiments, a time period during which the action should be performed is identified based on the identified status change. For example, the status change may indicate that a person is about to leave the room in which the gaming console 302, 402 is located, and therefore the action should be performed in a relatively short time period.

[0037] FIG. 5 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure. In a manner similar to the environment 200 discussed in connection with FIG. 2, the environment 500 includes a smart TV 502, a remote control device 504 in communication with the smart TV 502, and a camera 506. FIG. 5 depicts a first scenario 522 in which the remote control device 504 is in an unused state and a second scenario 524 in which the remote control device 504 is in use. The smart TV 502 generates a content item output that is displayed via a display of the smart TV 502. A user interacts with the smart TV 502 via the remote control device 504. In this example, the remote control device 504 is a wireless controller that communicates with the smart TV 502 via infrared. The movement or orientation of the remote control device 504 may be tracked in a manner similar to that described with reference to the controller 106 depicted in FIG. 1. Additionally, one or more thresholds similar to or the same as those described with reference to FIG. 1 may be utilized to determine whether a change in the status of the remote control device 504 has occurred.

[0038] A threshold is identified (508) based on the status of the controller 504 over a period of time. When the user lifts the controller 504 in a second scenario 524, a change in status is identified (510) based on the threshold. Upon identifying the change in status, an indication of the change in status is received (512). Upon receiving the indication of the change in status, an action to take is determined and subsequently taken. In this example, three actions 514a, 514b, 514c to take are depicted, however, in some examples, it is envisioned that only one or two of the actions 514a, 514b, 514c are taken on the smart TV 502. The first action 514a includes generating and displaying an overview, shortened, or "last gasp" advertisement on the smart TV 502. Instead of displaying the entire advertisement, a shortened advertisement is generated for display. For example, if the original advertisement is 25 seconds long, the advertisement may be replaced with a logo and / or message 516 and jingle regarding the product being advertised. The second action 514b includes generating and displaying an advertisement summary 518. Instead of displaying the advertisements, the advertisement summary 518 is generated for display. For example, if six advertisements are scheduled to be displayed during a commercial break, six advertisement summary 518, e.g., still images, are generated for output. When the change in status of the remote control device 504 indicates that it has been lifted, this may indicate that the user is about to change channels and / or fast forward through advertisements. By displaying the advertisement or advertisement summary, the advertisement or advertisements may still affect the user before any channel change or fast forward occurs. In some implementations, the advertisement or advertisement summary may be generated in the smart TV 502. In other implementations, the advertisement or advertisement summary may be generated in a server remote from the smart TV 502 and transmitted to the smart TV 502 over a network, such as the Internet.The third action 514c includes generating a user interface element 520 for receiving the feedback. For example, a status box with the text "Is this commercial relevant to you?" may be displayed along with user interface elements for "yes" and "no". In some implementations, the user interface element may be generated at the smart TV 502. In other implementations, the user interface element may be generated at a server remote from the smart TV 502 and transmitted to the smart TV 502 over a network such as the Internet. The advertisement may be based on the media item in that the advertiser may indicate that the advertisement may be displayed in a commercial break associated with the media item. Any of the first, second, and second actions 514a, 514b, 514c may have one or more audio components associated with it.

[0039] Any of the actions 514a, 514b, 514c may be combined in any suitable manner. Additionally, other actions may occur and be performed on the smart TV 502. Other actions include, but are not limited to, preloading any advertisements and / or elements of the media item and performing software updates associated with the smart TV 502 (e.g., updating software and / or components associated with a video game). In some implementations, the action or actions may be performed on a secondary computing device associated with the controller and / or smart TV 502. For example, a secondary computing device such as a remote control device 504 and / or a smart watch may include a display on which advertisements are displayed. In this implementation, the advertisement may be transmitted from the smart TV 502 via local wireless means. In other implementations, the advertisement may be received from a server over a network, such as the Internet, in response to a communication sent from the smart TV 502 to the server over the network. In another example, the secondary computing device may be a smart speaker and the action may include playing audio on the smart speaker, e.g., advertising audio and / or a voice message indicating the status of the smart TV 502, such as "TV is turned on." In some examples, a user may provide audio input to the smart speaker to perform an additional action, such as turning off the smart TV. In some examples, a time period during which the action should be performed is identified based on the identified status change. For example, the status change may indicate that a person is about to leave the room in which the smart TV 502 is located, and therefore the action should be performed in a relatively short time period.

[0040] FIG. 6 illustrates another example environment in which actions are performed in response to a change in the status of an input device, according to some embodiments of the present disclosure. In a manner similar to the environments 200, 500 discussed in connection with FIGS. 2 and 5, the environment 600 includes a smart TV 602, a remote control device 604 in communication with the smart TV 602, and a camera 606. FIG. 6 illustrates a first scenario 618 in which the remote control device 504 is in an unused state, and a second scenario 620 in which the remote control device 504 is in use. The smart TV 602 generates a content item output that is displayed via a display of the smart TV 602. A user interacts with the smart TV 602 via the remote control device 604. In this example, the remote control device 604 is a wireless controller that communicates with the smart TV 602 via infrared. The movement or orientation of the remote control device 604 may be tracked in a manner similar to that described with reference to the controller 106 depicted in FIG. 1. Additionally, one or more thresholds similar to or the same as those described with reference to FIG. 1 may be utilized to determine whether a change in the status of the remote control device 604 has occurred.

[0041] A threshold is identified (608) based on the status of the controller 604 over a period of time. When the user lifts the controller 604 in a second scenario 620, a change in status is identified (610) based on the threshold. Upon identifying the change in status, an indication of the change in status is received (612). Upon receiving the indication of the change in status, an action to be taken is determined and subsequently taken. In this example, the action to be taken includes accessing a user profile 614 and generating personalized advertisements 616 for display on the smart TV 602. The user profile may be associated with a single user and / or an entire household. The user profile may be stored locally on the smart TV 602 or received from a server remote from the smart TV 602 over a network such as the Internet. Any of the other actions described herein (e.g., actions 316, 316b, 416a, 416b, 514a, 514b, 514c) may be personalized based on the user profile.

[0042] FIG. 7 illustrates an example environment in which a change in the status of an input device is identified according to some embodiments of the present disclosure. The environment includes a smart TV 700, a camera 702, a remote control device 704, and a virtual border 706. The camera 702 communicates with the smart TV 700 and receives captures with the remote control device 704. A trained machine learning algorithm uses machine vision to identify the remote control device 706. The algorithm also applies the virtual border 706 to the remote control device 704. In determining the threshold, the machine learning algorithm may determine how often an edge of the remote control device crosses the virtual border 706 over a period of time. For example, if the remote control device 704 is placed on a table, the remote control device may be completely still and may not cross the border at all, however, in a second example, if the remote control device is placed on a user's lap, the remote control device may move around slightly but still be substantially still. In this second example, the edge of the remote control device may cross the border 706 occasionally and / or to a small extent. In this manner, a threshold can be identified based on the status of the input device. In a first example, the threshold may be that no portion of the remote control device 704 crosses the virtual boundary. In a second example, the threshold may be based on the frequency and / or extent to which the remote control device 704 crosses the virtual boundary. When the user lifts the remote control device, a much larger portion of the remote control device will cross the virtual boundary 706 and / or no portion of the remote control device may be within the virtual boundary 706. Because this movement is greater than the identified threshold, a change in the status of the remote control device is identified.

[0043] In a similar manner, a threshold may be identified via an IMU of an input device. In an embodiment where the input device is a controller for a gaming console, the output of the IMU may indicate that the controller is physically moving in three-dimensional space and / or a profile is generated that is associated with user input, for example, via a button on the controller. A threshold may be identified for the output of the IMU over a period of time. In this embodiment, the threshold may be based on the amount the controller moves, such that a change in status above the threshold indicates that the controller is stationary or substantially stationary. When the user puts the controller down, the output of the IMU becomes stationary or substantially stationary, which is greater than the identified threshold, and a change in the status of the controller is identified.

[0044] 8 shows a block diagram representing components of a computing device and data flows therebetween for enabling actions to be performed in response to changes in the status of an input device, according to some embodiments of the present disclosure. A computing device 800 as discussed above (e.g., computing devices 102, 202, 302, 402, 502, 602, 700) comprises input circuitry 804, control circuitry 808, and output circuitry 830. The control circuitry 808 comprises control circuitry and memory circuitry that may be based on any suitable processing circuitry (not shown) and may be located on a single integrated circuit or may be discrete components and processing circuitry. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc., and may include multi-core processors (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some embodiments, the processing circuitry may be distributed across multiple separate processors or processing units, such as multiple processing units of the same type (e.g., two Intel Core i9 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor), and / or a system on a chip (e.g., a Qualcomm Snapdragon 888). Some control circuitry may be implemented in hardware, firmware, or software.

[0045] Input is received (802) by input circuitry 804. Input circuitry 802 is configured to receive input associated with a computing device. For example, this may be via an infrared controller, Bluetooth and / or Wi-Fi controller, touch screen, keyboard, mouse, and / or microphone of computing device 800. In other examples, this may be via gestures detected via an augmented, mixed, and / or virtual reality device. In another example, the input may comprise instructions received via another computing device. Input circuitry 804 transmits the user input to control circuitry 808 (806).

[0046] The control circuitry 808 comprises a media item generating module 810, a threshold identification module 814, a status identification module 818, a status indication receiving module 822, and an action generating module 826. The control circuitry 808 is in communication with output circuitry 830. The output circuitry 830 comprises an action taking module 832. An input is transmitted (806) to the media item generating module 810 where a media item is to be generated for output. An indication that a media item is being generated for output is transmitted (812) to a threshold identification module 814 where a threshold is identified based on the status of the input device over a period of time. For example, it may be determined that the input device is stationary or substantially stationary for a period of time, and the threshold may be based on identifying a threshold that indicates that the input device has changed from a substantially stationary state to a moving state. In another example, the threshold may be identified conversely and the input device may be determined to be in motion for at least a substantial portion of the time period, and the threshold may be based on identifying a threshold that indicates the input device has changed from a moving or substantially moving state to a substantially stationary state. In another example, the threshold may be associated with a particular inertial movement profile of the input device, for example, an inertial movement profile that is typically associated with the input device being lifted and / or placed on a surface. The identified threshold is transmitted (816) to a status identification module 818, where a change in the status of the input device that exceeds the threshold is identified, for example, that the input device has been lifted and / or placed down. Upon identifying that the change in the status of the input device exceeds the threshold, an indication is transmitted (820) to a status indication receiving module 822. In some examples, this may include receiving an indication from the input device itself. In other examples, this may include one component (e.g., a hardware and / or software component) of the computing device receiving an indication from another component of the computing device.An indication is transmitted 824 to an action generation module 826 where an action is generated based on the media item being generated for output. An indication of the action is transmitted 828 to output circuitry 830 where the action is performed by an action implementation module 832.

[0047] 9 shows a flowchart of illustrative steps involved in enabling actions to be performed in response to a change in the status of an input device, according to some embodiments of the present disclosure. Process 900 may be implemented on any of the computing devices mentioned above (e.g., computing devices 102, 202, 302, 402, 502, 602, 700). In addition, one or more actions of process 900 may be incorporated into or combined with one or more actions of any other process or embodiment described herein.

[0048] At 902, a media item is generated for output and at 904, it is determined whether the media item is a video game or a content item. If it is determined that the media item is a video game, at 906, a threshold is identified based on movement of the input device. At 908, it is identified whether a lack of movement of the input device exceeds a threshold (i.e., the input device remains relatively stationary). At 910, an indication of a change in status is received and at 912, an action to take is identified. At 914, it is determined whether a user profile is available. If a user profile is available, at 916, the user profile is accessed. The process proceeds from step 914 or 916 to 918, where it is identified whether the video game should be paused. If the video game should be paused, the video game is paused at 920. The process proceeds from step 918 or 920 to perform at least one action 922, 924, 926, 928, 930 identified in 912. At 922, the action to perform is to generate and / or preload an advertisement for output. The advertisement may be visual only, audio only, and / or a combination of both. At 924, an element is introduced and / or replaced in the video game. For example, a character holding a drink can may have an unbranded drink can replaced with a branded drink can. At 926, a replay is generated. For example, if a multiplayer video game is paused, a summary of the missed action may be generated for output. At 928, a character's characteristics may be changed, for example, a character may be made invulnerable in the multiplayer game while the video game is paused. At 930, an update may be performed, such as a video game update. Any of the actions 922, 924, 926, 928, 930 may be performed alone or in combination with any of the other actions 922, 924, 926, 928, 930.In some embodiments, one or more actions may be repeated.

[0049] If it is determined that the media item is a content item, then at 932, a threshold is identified based on the input device being stationary. At 934, it is identified that the movement of the input device exceeds a threshold (i.e., the input device is lifted). At 936, an indication of the change in status is received and at 938, an action to perform is identified. At 940, it is determined whether a user profile is available. If a user profile is available, at 942, the user profile is accessed. The process proceeds from step 940 or 942 to perform at least one action 944, 946, 948 identified at 938. At 944, the action to perform is to generate an overview of the multiple advertisements. For example, if six advertisements are scheduled to be displayed during a commercial break, an overview of the six advertisements, e.g., still images, is generated for output. At 946, the action to perform is to generate a user interface element for receiving feedback. For example, a status box with the text "Does this commercial relate to you?" may be displayed along with user interface elements for "Yes" and "No." At 948, a shortened advertisement is generated for display. For example, if the original advertisement is 25 seconds long, the advertisement may be replaced with a logo and jingle related to the product being advertised. If a user profile is accessed, actions may be based at least in part on the user profile. For example, the advertisement may be customized based on the user profile. Any of the actions 944, 946, 948 may be implemented alone or in combination with any other of the actions 944, 946, 948. In some embodiments, one or more actions may be repeated.

[0050] The process described above is intended to be illustrative and not limiting. Those skilled in the art will understand that the steps of the process discussed herein may be omitted, modified, combined, and / or rearranged, and any additional steps may be implemented without departing from the scope of the present disclosure. More generally, the above disclosure is meant to be illustrative and not limiting. Only the following claims are meant to set boundaries on what the invention includes. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and that the flow charts or examples related to one embodiment may be combined with any other embodiment in a suitable manner, performed in a different order, or performed in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and / or methods described above may be applied to or used in accordance with other systems and / or methods.

Claims

1. It is a method, In computing devices, generating media items for output, Identifying a threshold, wherein the threshold is based on the status of the input device over a certain period of time, and the input device is associated with the computing device. To identify a change in the status of the input device that exceeds the aforementioned threshold, Receiving an indication of a change in the status of the input device, In response to receiving the aforementioned indication, perform an action based on the media item. Methods that include...

2. The method according to claim 1, wherein the media item is a video game, and performing the action further includes pausing the video game and generating an advertisement for output.

3. The method of claim 2, further comprising generating an advertisement for output, and modifying elements of the video game based on the advertisement.

4. The aforementioned media item is a video game, The change in the status of the input device is a first change in status, Implementing the aforementioned action further means Identifying a second change in the status of the input device, Based on the gameplay of the video game between the first change in status and the second change in status, content items are generated for output. The method according to claim 1, including the method described in claim 1.

5. The method according to claim 1, wherein identifying a change in the status of the input device further includes identifying the movement of the input device.

6. Implementing the aforementioned action further means Identifying that an ad is being generated for output, The means of generating a user interface element for output, wherein the user interface element is configured to allow input to the advertisement to be received. The method according to claim 1, including the method described in claim 1.

7. The user profile is associated with the computing device, The method according to claim 1, wherein performing the aforementioned action further includes generating advertisements for output based on the user profile.

8. A method, In a display device, to provide an interactive media item for display, In the first time period, the input received at the interface of the input device associated with the display device is detected, Based on the profile associated with the display device, identify a threshold period related to the input device being in an idle state, The process involves comparing the first time and the second time to determine the period elapsed since the input was detected, and that no input was detected during that period. Based on the determination that the aforementioned period has exceeded the threshold period, an action based on the interactable media item is taken. Methods that include...

9. A system comprising a control circuit network configured to perform the method described in any one of claims 1 to 8.

10. A non-transient computer-readable medium having instructions encoded thereon, wherein, when executed by a control circuit network, the instructions cause the control circuit network to perform the method according to any one of claims 1 to 8.

Images