Method for enabling user-defined content-aware behavior - Patents.com

Abstract

A method performed by a viewing device includes receiving a request for an action to be performed by the viewing device. The request includes trigger criteria for performing the action. The viewing device receives content from a content provider, the content including content-related metadata. Detection is performed of a condition that the trigger criteria for the action to be performed is met. As a result, the viewing device performs the requested action. Detection may be accomplished by the viewing device or by the service provider.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 161,767, filed March 16, 2021, which is incorporated herein by reference in its entirety for all purposes.

[0002] FIELD OF THEINVENTION The present principles relate to communications from a digital television device, and in particular to a method for issuing content-aware requests to the device so that the device performs content-aware actions on behalf of a user. [Background technology]

[0003] Devices capable of displaying broadcast or streamed multimedia content currently lack the ability for users to issue on-the-fly content-aware requests to these devices so that the devices perform content-aware actions on the user's behalf. One exemplary use case illustrating the need for content-aware requests is a husband and wife couple watching television (TV) together. The husband is an avid football fan, while the wife is merely a casual fan. The husband's favorite soccer team is currently playing and the game is being televised live. The husband and wife watch the start of the game together, but as the husband's team is falling behind, the wife begins to lose interest and asks if there is something else to watch. The husband complies, but periodically switches back and forth between the game and his wife's show to make sure his team doesn't make a comeback and the game becomes interesting again. However, the wife eventually becomes annoyed with the switching back and forth and takes the remote away from her husband so he can no longer check back on the game. The husband's team ends up coming back to win the game, but unfortunately, the husband ends up losing sight of the come-from-behind victory.

[0004] The above example use cases illustrate the need to monitor the content of one program while watching another program without using picture-in-picture or channel changing to actively inspect the content and determine if the monitored channel is suitable for viewing. Summary of the Invention

[0005] This Summary is provided to introduce selected concepts in a simplified form as a prelude to the more detailed description that is presented later. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] In one embodiment, a method performed by a device includes receiving a request for an action to be performed by the device. The request includes trigger criteria including a content-based event occurring within a content selection. The device transmits the trigger criteria to a content provider. A receiver receives metadata associated with the content-based event. Detection of the content-based event occurs based on the received metadata. If the trigger criteria is determined to be met, the device performs the requested action.

[0007] Further features and advantages will become apparent from the following detailed description of illustrative embodiments, which proceeds with reference to the accompanying drawings. The drawings are intended to illustrate the concepts of the disclosure and are not necessarily the only possible configurations for illustrating the disclosure. Features of various drawings can be combined unless otherwise specified. [Brief description of the drawings]

[0008] The foregoing summary, as well as the following detailed description of illustrative embodiments, will be better understood when read in conjunction with the accompanying drawings, included by way of example, and not by way of limitation, with respect to the present principles, in which like numerals represent like elements and in which: [Figure 1]1 depicts a diagram showing the functional entities of the present disclosure; [Diagram 2] 1 illustrates a message exchange and function diagram for establishing default settings. [Diagram 3] 1 illustrates a message exchange and functionality diagram for configuring content-aware operations. [Figure 4] 1 depicts a functional diagram showing a local content inspection and action process. [Diagram 5] 1 illustrates a functional diagram of a content provider assistance process. [Figure 6] 1 depicts separate content notification provider functionality. [Figure 7] 1 depicts a system level embodiment of user-defined content-aware behavior functionality. [Figure 8] 1 depicts an interface for a system level embodiment of user-defined content-aware behavior. [Figure 9] 1 depicts an exemplary flow diagram of a method according to an aspect of the present disclosure. [Figure 10] 1 illustrates an exemplary block diagram of a content aware processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following description of various illustrative embodiments, reference is made to the accompanying drawings, which form a part hereof, and which show, by way of illustration, how the various embodiments may be practiced. It is to be understood that other embodiments may be utilized and structural and functional modifications may be made without departing from the scope of the present principles.

[0010] This disclosure defines methods, techniques, and devices to address the need to monitor the content of one program while watching another program, as well as details for devices (e.g., digital television (DTV), phones, tablets, personal computers (PC), augmented reality head-mounted devices (AR HMDs), etc.) capable of playing broadcast or streamed multimedia content (e.g., movies, shows, live sports, etc.) to support advanced user-defined content-aware operations. This disclosure defines features that enable devices (e.g., DTVs, phones, tablets, PCs, etc.) capable of playing broadcast or streamed multimedia content to support the ability for a user to issue customized on-the-fly content-aware requests to the device so that the device performs content-aware operations on the user's behalf. For example, if a user is watching a televised game involving his or her favorite soccer team and his or her team falls behind and the user loses interest in the game, the user can switch to watching something else, but if his or her favorite team is back and vying to win the game, the user can also issue a request to the device to switch back to the game. That way, the user doesn't have to manually switch back and forth between games to determine if something interesting has happened.

[0011] Characteristics of a multimedia device that supports content-aware functionality capable of content-aware operation may include one or more of the following capabilities: A request to configure a customized content-aware behavior is received from a user. b. Process the request by analyzing the content aware behavior from the user. d. Determine whether the requested content-aware operation is supported by the device. e. Remember content-aware actions from the user. f. Return a response to the user indicating that the user's content aware behavior was accepted by the device. g. Monitoring the multimedia content and comparing the multimedia content with the user's requested content-aware behavior. h. Detect when a content aware trigger criteria defined within a content aware action is satisfied by the monitored multimedia content. i. Execute one or more content-aware actions defined within the content-aware behavior (eg, change to the channel on which the multimedia content is being broadcast).

[0012] The received content-aware action request may be issued by the user to the device in one or more of the following ways: a. User initiated voice (or gesture) commands, including customized content-aware actions; or b. A request that is configured via an app on the user's smart device and issued to the device, including customized content-aware behavior; or c. a series of one or more interactive device menu selections made by a user while interacting with the device (e.g., via a remote control) to configure the device with customized content-aware behavior; or d. If biosignal data is available to the user, actions may be automatically created based on the user's measured response to the currently viewed content (e.g., "booing" when one team score may result in the creation of an action to notify the user when the other team does something positive). This may be a "suggested" action made available to the user that the user may review before being implemented by the system, or e. If a user manually switches to a channel multiple times, the system may create an action to notify the user whenever something of interest is occurring on that channel. This may be a "suggested" action made available to the user that the user may review before being implemented by the system.

[0013] Content-aware operations may include, but are not limited to, one or more of the following elements: One or more multimedia content identifiers (eg, name, channel, scheduled time slot, etc.) for a particular content selection of interest (eg, a particular movie, show, or sporting event, etc.). b. One or more multimedia content selection filters that identify the types of content in which the user is interested (eg, favorite content genres, favorite sports teams, favorite actors, etc.). c. Content-aware trigger criteria such as, but not limited to: i. A specified content-based event that occurs within the multimedia content selection (e.g., the beginning / end of a timeout in a sporting event, a change in the score in a sporting event, a performance by a particular singer at an awards show, a particular scene in a movie, the start of the final Japardi round, etc.). ii. The start / end of a commercial advertisement associated with a multimedia content selection. d. A list of one or more content-aware actions that the device should perform if / when the content-aware trigger criteria is met. e. Content-aware behavior preferences that apply to a user, including, but not limited to: i. Default settings that are applied to a user's content-aware action requests so that the user does not have to specify details in every request that they issue to the device. For example, default screen settings may be available to the user if / when a content-aware action is performed by the device if / when the user does not specify the screen on which the user wants the device to display the content. f. A schedule that defines when the device should perform content-aware operations. g. Settings to enable and / or disable the device from performing content-aware actions. h. An identifier for the user, an identifier for the user's device, and / or an identifier for the application hosted on the user's device to which the content and / or notification is sent by the device.

[0014] A user's content aware actions may be stored in places such as, but not limited to, the following: A device's local storage (eg, memory). b. A peer device directly attached to the device (eg, a set-top box (STB) communicatively coupled to a display device). c. Services provided within the network and interconnected to devices (e.g., cloud servers, gateways, etc.).

[0015] The response generated by the device in response to receiving a content-aware action request may be directed to one or more of the following: A pop-up window that appears on the primary device screen. b. A pop-up window displayed on the screen of the user's secondary device. c. A text message sent to the user's secondary device. d. A message sent to an application hosted on a secondary device of the user. e. Audio alerts played through the device sound system.

[0016] Monitoring the multimedia content to detect whether content-aware trigger criteria have been met may include content-aware actions performed by the device, such as, but not limited to, one or more of the following: a. Screening received multimedia content using content monitoring screening techniques such as: i. Processing metadata piggybacked onto the received multimedia content, including additional information of relevance to the content and content-aware trigger criteria, and detecting whether the criteria have been met. The metadata may be generated by or associated with the multimedia content provider, or may be provided by a third-party service. ii. Analyzing the received multimedia content in real time using deep multimedia content screening algorithms to detect attributes of the multimedia content that match content-aware trigger criteria specified by a user. b. Sending the user's content aware trigger criteria to other entities in the system (e.g., a content provider's content notification provider) so that other entities can monitor and detect if / when the criteria are met. c. Subscribe to other entities in the system (eg, a content provider's content notification provider) to be notified when the user's content aware trigger criteria are met.

[0017] Receiving notifications from another entity in the system (e.g., a content provider's content notification provider) if / when a user's content aware trigger criteria is met.

[0018] Synchronizing a notification received from another entity in the system that a trigger criterion for a content-aware operation has been met with multimedia content received by the device.

[0019] Multimedia content aware actions performed by a device in response to detecting that a multimedia content aware trigger criterion has been met may include, but are not limited to: a. Send a message to the user informing them that the criteria of interest have been met. b. Ask the user if he / she wishes the device to proceed to playing multimedia content applicable to the criteria. c. Change channels and play content on the device that meets the criteria. d. Play / display the multimedia content meeting the criteria on the user's secondary device. e. Recording / buffering a multimedia content clip within a time window before and / or after a content aware trigger criterion is met (eg, playing a game resulting in a score change). f. Creating / appending multimedia content clips onto an aggregated content collection (eg, a user's personal content highlight reel). g. Synchronizing content-aware behavior generated by another entity in the system with multimedia content received at the device so that actions taken by the device in response to a notification occur at the optimal time (e.g., so that the user does not miss content they want to watch). h. Resolving conflicts between multiple user-defined actions that may arise by doing one or more of the following: i. Prioritizing certain users and certain user actions over other users based on the user's profile (ie, dropping lower priority content aware behaviors and actions). ii. Recording / buffering multimedia content, content-aware operations and actions so that they can be delayed and serialized after other multimedia content, content-aware operations and actions that have been performed and not missed.

[0020] 1 depicts a functional diagram 100 depicting one or more entities for dealing with supporting devices 115 (e.g., DTV, phone, tablet, PC, AR HMD) capable of playing broadcast or streamed multimedia content (e.g., movies, shows, live sports, etc.) to support advanced user-defined content-aware behavior. Below is an overview of the proposed user-defined content-aware behavior functionality.

[0021] In FIG. 1, the user 110 may utilize a remote device (not shown) to interact with the display device 115 or may interact directly with the display device 115. The content provider 120 may operate to interface with the display device 115. In 101, an interaction functionality (remote control or direct interaction) is provided to allow the user 110 to pre-configure the display device 115 with the user's default settings that the device applies to any future content-aware action requests that the user 110 may issue to the display device 115. This can simplify the interaction between the user and the device by reducing the amount of information that needs to be conveyed to the device within each individual content-aware action request that the user issues to the device. For example, if the user wants the device to display by default broadcast or streamed multimedia content associated with the content-aware action on the device screen if / when the trigger criteria for the content-aware action are met, the user can configure this as the content-aware action default setting. In contrast, if a user wants the device to display broadcast or streamed multimedia content associated with content-aware operation on the screen of a secondary device (e.g., the user's smartphone) by default, the user can configure this as the content-aware operation default setting.

[0022] 1, a user 110 may issue a request to pre-configure a display device 115 (e.g., a DTV) with user-defined default settings that are applicable to subsequent content-aware operation requests that the user may issue to the device. The default settings are stored by the device 115 and applied to subsequent content-aware operation requests received from the user 110. As a result, this content-aware operation default setting capability can reduce the amount of information that a user needs to include in any subsequent content-aware operation requests that the user issues to the device. See below for further details.

[0023] At 102 in FIG. 1, there may be interactive functionality to allow a user to issue requests to a display device 115 to cause the device to perform customized content-aware actions on the user's behalf. The requests may include user-defined trigger criteria and actions that have relevance to broadcast or streamed multimedia content of interest to the user. For example, a request to a DTV to switch to a televised political conference when a user-specified person of interest begins speaking. At 102 in FIG. 1, a user 110 issues individual requests to a device 115 (e.g., a DTV) to cause the device to perform user-defined content-aware actions on the user's behalf.

[0024] 1, at 103b, there may be functionality within the display device 115 to enable the display device 115 to detect if / when a trigger criterion defined by the user's content-aware behavior has been met. Note at 103a, this functionality may optionally involve the device interacting with another entity in the system (e.g., the content provider's notification service 120) to assist the device in detecting this trigger criterion. For example, detecting when a user-specified person of interest begins speaking at a televised political meeting that is broadcast or streamed to the device.

[0025] At 103a in FIG. 1, the device 115 may optionally interface with a content provider and / or a content notification provider 120 to assist in detecting if / when a trigger criterion associated with a content-aware action has been met. For further details, see below. At 103b, the device 115 detects that a trigger criterion of a content-aware action has been met. For example, the device may detect that a game score has changed or that a user-specified scene of interest in a movie has been reached. This step / function may be performed exclusively by the device 115 itself using content-aware action functionality hosted locally by the device 115. Alternatively, the device may rely on information received from the content provider and / or the content notification provider 120 (i.e., step 3a) to enable the device 115 to detect that a content-aware trigger criterion specified by the user has been met.

[0026] 1, in 104, functionality may be present to enable the display device 115 to execute a content-centric action defined for a content-aware operation if / when a corresponding trigger criterion of the content-aware operation is met, e.g., switching the channel to one on which a political conference is being televised.

[0027] 1, the device 115 executes one or more user-defined actions of the content-aware operation as a result of detecting that the trigger criteria for the content-aware operation have been met. For example, the device may switch a display to show content of interest to the user, send a notification about the content of interest to a secondary device (e.g., a smartphone) of the user, and / or record / buffer the content of interest for later viewing.

[0028] It should be noted that while many of the embodiments described in this disclosure define content-aware operational features in relation to display devices such as DTVs, those skilled in the art will recognize that these proposed concepts may also be applicable to other types of consumer electronic devices such as, but not limited to, smartphones, tablets, PCs, and the like.

[0029] Content Aware Behavior Default Settings FIG. 2 depicts a message exchange and functional diagram for establishing default settings. As shown in FIG. 2, a user 210 may configure a device 215 with default settings that are applicable to subsequent content-aware action requests that the user 210 may issue to the device 215. These default settings include, but are not limited to, content-aware action settings, such as those defined in Table 1. These default settings may be defined such that the user 210 does not have to specify values ​​for these elements in each individual content-aware action that the user issues to the device 215. Instead, the defaults may be pre-configured by the user 210, stored by the device 215, and applied to each content-aware action issued by the user 210. A user may choose to override the default setting of an element by explicitly including the element in an individual content-aware action request that the user issues to the device 215.

[0030] In 201 of FIG. 2, a user 210 issues a request to a device 215 to create, retrieve, update, or delete user-defined content-aware behavior default settings, which may include information elements defined in Table 1. The user may issue this request to the device via different methods, such as a voice command from the user that includes the content-aware behavior default settings, a request that is configured and issued to the device via an app hosted on another device (e.g., the user's smartphone) and includes the content-aware behavior default settings, or a series of one or more interactive menu selections made by the user while interacting with the device (e.g., via a TV remote control) to pre-configure the device with the content-aware behavior default settings.

[0031] In 202 of FIG. 2, the device 215 receives and processes the request by verifying whether the content-aware action default settings are supported by the hosting device's local content-aware action service. The device 215 may verify support by checking whether the specified settings are compatible with the device's supported content-aware action service capabilities. For example, whether the device supports actions such as issuing a notification to the user or recording a content segment of interest when / if the content-aware notification criteria are met. If the requested action cannot be supported locally, the device may forward the request to the content / notification provider 220 to see if they can support the action. In this case, some actions may be performed locally on the device 215 and some actions may depend on the assistance of a remote entity, such as the content provider 220. The device 215 may maintain a record of which actions are performed by which entities.

[0032] At 203 in FIG. 2, the device 215 may optionally send one or more requests to create, retrieve, update, or delete content-aware behavior default settings to a content provider supporting a remote content-aware behavior service and / or to a content notification provider 220. Within the request, the device 215 may include information elements defined in Table 1. For example, the device 215 may share the user's content-aware behavior default settings with the content notification provider 220 and use these settings to customize content-based notifications sent to the device 215. For example, the content notification provider 220 may send only notifications that are applicable to content of interest to the user of the device and that include information applicable to the user's defined content-based criteria.

[0033] 2, the content provider and / or content notification provider 220 receives and processes the request by checking whether the specified settings are compatible with their content-aware behavior service capabilities, and if so, a content-aware behavior default setting is created, retrieved, updated, or deleted.

[0034] 2, the content provider and / or content notification provider 220 returns a response to the device 215 confirming whether the user's content aware behavior default settings were successfully created, retrieved, updated, or deleted. Within this response, the information elements defined in Table 1 may be included and updated based on the capabilities of the content provider and / or content notification provider 220.

[0035] At 206 in FIG. 2, if the device 215 determines that user-defined content-aware behavior default settings can be supported, the device creates, retrieves, updates, or deletes default settings for the user 210. Any created and updated default settings can be accessed later if / when the device 215 receives a content-aware behavior request from the user 210. The device 215 can store the user's content-aware behavior default settings in locations such as, but not limited to, its local storage (e.g., memory), attached peer devices (e.g., DTVs or set-top boxes (STBs) with connected displays, not shown in FIG. 2), services provided in a network and interconnected to the device (e.g., cloud servers, gateways, etc., not shown in FIG. 2). The exemplary device 215 shown in FIG. 2 is a DTV, which can include, but is not limited to, DTVs, smartphones, tablets, laptops, smartwatches, smart glasses, AR HMDs, STBs with connected displays, etc.

[0036] At 207 in FIG. 2, the device 215 responds to the user 210 indicating whether the request to create, retrieve, update, or delete default settings for content-aware behavior was accepted or rejected by the device. Within this response, information elements as defined in Table 1 may be included. The values ​​of these elements may match those specified in the request or may be updated to take into account changes (e.g., override values) that the device, content provider, and / or content notification provider 220 make to these elements. The response may be directed to one or more of the following: A pop-up window will appear on the screen of your device. b. An audio alert played through the device's sound system. c. A pop-up window displayed on the screen of the user's secondary device. d. A text message sent to a user's secondary device. e. Messages sent to an application hosted on a secondary device of the user.

[0037] Content-aware behavior request FIG. 3 depicts a message exchange and functional diagram useful for configuring content-aware actions. As shown in FIG. 3, a user 310 can issue individual requests to the device 315 to cause the device 315 to perform user-defined content-aware actions on the user's behalf. These requests can take the form of requests issued by the user in an ad-hoc / on-the-fly manner (e.g., "Hey TV, show me when the Eagles score today"). These actions can be short-term and only applicable to a duration defined by the content itself (e.g., the duration of a currently airing movie or show). Alternatively, the user 310 can configure the device 315 with more persistent or recurring content-aware actions that the device can store and use to perform recurring content-aware actions on the user's behalf if / when the action's trigger criteria (e.g., "Hey TV, record every time the Eagles score") are met. In this case, the device can store a schedule of repeating / recurring content-aware actions.

[0038] [Table 1-1]

[0039] [Table 1-2]

[0040] At 301 in Figure 3, a user 310 issues a request to a device 315 to create, retrieve, update, or delete a user-defined content-aware action, which may include information elements such as, but not limited to, those defined in Table 2. The user may issue this request to the device via different methods, such as a voice command from the user that includes the content-aware action, a request that is configured and issued to the device via an app hosted on another device (e.g., the user's smartphone) and includes the content-aware action, or a series of one or more interactive menu selections made by the user while interacting with the device (e.g., via a TV remote control) to configure the device with the content-aware action.

[0041] At 302 in FIG. 3, the device 315 receives and processes the request by verifying whether the type of content-aware action the user is requesting is supported by the hosting device's local content-aware action service. The device 315 may verify support for the type of content-aware action by checking whether the device has access to the specified content of interest and whether the specified trigger criteria and actions defined within the content-aware action are supported by the device content-aware action service. Before verifying support for the request, the device may first apply the user's default content-aware action settings to the request, which may add and / or update information elements of the request.

[0042] 3, the device 315 may optionally send one or more requests to content providers and / or content notification providers 320 that support remote content-aware operation services to see if they can support the user's requested type of content-aware operation. Within this request, the device may include information elements such as, but not limited to, those defined in Table 2. When sending these information elements, it may not be necessary to send all elements within the request to the content provider 320. For example, only the applicable content and trigger criteria may be required.

[0043] 3, the content provider and / or content notification provider 320 receives and processes the request by verifying whether the type of content-aware operation requested by the user is supported. Before verifying support for the request, the content provider and / or content notification provider may first apply the user's default content-aware operation settings to the request, which may add missing elements that are not present in the request.

[0044] 3, the content provider and / or content notification provider 320 responds to the device 315 whether it can support the type of content-aware operation requested by the user. Information elements such as, but not limited to, those defined in Table 2 may be included in the response.

[0045] At 306 in FIG. 3, if the device 315 determines that the requested type of content-aware operation is supported, the device creates, retrieves, updates, or deletes information stored locally by the device regarding the user's content-aware operation. Once created / updated, the stored information of the content-aware operation is accessed and used by the device to monitor for detection of the user's content-aware trigger criteria and to determine which content-aware action to perform if / when the content-aware operation is triggered. The device 315 may store the user's content-aware operation in locations such as, but not limited to, its local storage (e.g., memory), attached peer devices (e.g., displays coupled to a DTV or set-top box not shown in FIG. 3), services provided within a network and interconnected to the device (e.g., cloud servers, gateways, etc. not shown in FIG. 3). The exemplary device 315 shown in FIG. 3 is a DTV, which may include, but is not limited to, DTVs, smartphones, tablets, laptops, smartwatches, smart glasses, AR HMDs, STBs with connected displays, etc.

[0046] 3, the device 315 responds to the user 310 indicating whether the request to create, retrieve, update, or delete a content-aware action was accepted or rejected by the device. The response may be directed to one or more of the following: A pop-up window will appear on the screen of your device. b. An audio alert played through the device's sound system. c. A pop-up window displayed on the screen of the user's secondary device. d. A text message sent to a user's secondary device. e. Messages sent to an application hosted on a secondary device of the user.

[0047] [Table 2]

[0048] Autonomous processing of content-aware behavior by devices 4 depicts a functional diagram of local content inspection and local content-aware action processing according to an aspect of the disclosure. In FIG. 4, a device 415 may support a local content-aware action service that enables the device to autonomously process content-aware actions without requiring assistance from other entities in the system. A device 415 with this capability may monitor and compare incoming content against trigger criteria defined in locally stored content-aware actions, detect if the trigger criteria are met, and perform the action defined by the locally stored content-aware action.

[0049] At 401 in Figure 4, a process such as that shown in Figure 2 and / or Figure 3 may be performed to set up and configure content-aware operations. At 402, a content provider 420 may provide a user selected multimedia content (e.g., a movie, a TV show, a live sporting event, etc.) to a device 415. At 403, the device 415 receives multimedia content from the content provider 420. At 404, the user 410 may optionally configure a new content-aware operation or reconfigure an existing content-aware operation as described in Figure 2 or Figure 3.

[0050] In 405 of FIG. 4, the device 415 locally inspects the multimedia content to determine if the user's content aware trigger criteria are met. To perform this local inspection of the multimedia content, the device 415 may support deep multimedia content inspection functionality. In one embodiment, this deep multimedia content inspection functionality may be realized as a multimedia content inspection engine that can support different multimedia content inspection algorithms used to inspect the multimedia content of interest and detect if the specified criteria are met by the multimedia content. The multimedia content inspection engine may be policy-based such that it may be configured with inputs such as different types of multimedia content algorithms, multimedia content streams, multimedia content aware trigger criteria, etc. The device 415 may support the ability to configure this multimedia content inspection engine with content aware behavior default settings and / or information elements included in a request that the device receives from the user 410 to configure the device 415 with one or more content aware behaviors. The multimedia content inspection engine may be used by the device to inspect the multimedia content in real time to detect the occurrence of specific content-based events that match the events specified in the user's content aware trigger criteria. The example device 415 shown in FIG. 4 is a DTV, and may include, but is not limited to, DTVs, smartphones, tablets, laptops, smartwatches, smart glasses, AR HMDs, STBs with connected displays, etc.

[0051] At 406 of FIG. 4, the device 415 may detect that a user's content aware trigger criteria have been met. In one embodiment, the device 415 may detect the user's content aware trigger criteria via a multimedia content inspection engine as described at 405. For example, this detection may involve detecting content aware trigger criteria related to content-based events, such as, but not limited to, those defined in Table 1. For example, a timeout or period beginning / ending in a broadcasted sporting event, a score changing in a broadcasted sporting event, detection of a particular image or scene of interest in a show or movie, a particular show or movie starting, a show or movie featuring a favorite actor / actress starting, and / or a particular point of interest being reached.

[0052] Upon detecting that a user's content-aware trigger criteria have been met at 407 in FIG. 4, the device 415 may perform one or more content-aware actions, such as, but not limited to, one or more of the following: a. Send a message to the user informing them that the criteria of interest have been met. b. Ask the user if he / she wishes the device to proceed to playing multimedia content applicable to the criteria. c. Switch to playing / viewing the multimedia content on devices that meet the criteria. d. Play / display the multimedia content meeting the criteria on the user's secondary device. e. Recording / buffering multimedia content clips that meet criteria for later viewing and / or repeat viewing (eg, game play resulting in a score change). f. Creating / appending content clips onto an aggregated content collection (eg, a user's personal content highlight reel).

[0053] Content provider assists in content-aware behavior 5 is an example depiction of a functional diagram for content provider-assisted processing of content-aware operations. In FIG. 5, a device 515 may rely on another entity (e.g., a content provider 520) to provide content metadata in addition to the raw content. Information contained within the content metadata may be used to enable the device to more easily and / or extensively inspect the content to detect if / when user-defined content-aware trigger criteria have been met.

[0054] In 501 of Figure 5, the content aware behavior is configured as in Figure 2 or Figure 3. In addition, the device 515 may share the content aware behavior information of the user 510 with one or more content providers 520. The device 515 can selectively share the content aware behavior information of the user with only certain content providers, for example, only those content providers that the device 515 determines to be of interest to the user and to provide multimedia content that meets the user's content aware behavior criteria (i.e., content aware trigger criteria).

[0055] At 502 in FIG. 5, the content provider 520 examines the multimedia content to detect content-based events of interest. In one embodiment (according to option #1 shown in FIG. 5), the content provider 520 may identify expected events that are likely to be of interest to the user. For example, the content provider may rely on historical statistics of events that were of interest to the user for similar types of multimedia content currently being consumed by the user of the device. Based on this information and analysis, the content provider 520 may identify expected content-based events that are likely to be of interest for the multimedia content currently being consumed by the user of the device.

[0056] In a related second embodiment (according to option #2 shown in FIG. 5 ), the content-based events of interest may be defined by a current user who has configured content-aware default settings and / or one or more content-aware actions on the device 515 that are forwarded to the content provider 520. For example, content-aware trigger criteria may be detected that include content-based events of interest to the user.

[0057] Option #1 - Content providers generate content aware metadata. 5, content provider 520 generates content metadata based on the results of inspecting the multimedia content to detect content-based events of interest in 502. This content metadata is then piggybacked and transmitted along with the broadcast or streamed multimedia content that the content provider delivers to devices, such as device 515. The generated content metadata may include information about the content-based events detected by content provider 520. The content-based events in the content metadata may include one or more elements defined in Table 3.

[0058] At 504 of Figure 5, the device 515 receives multimedia content and content metadata from the content provider 520. At 505, the device 515 locally inspects the content metadata to determine whether the user's content aware trigger criteria have been met. To perform this local inspection of the content metadata, the device 515 may support content metadata inspection functionality that is used by the device to inspect the content metadata in real-time to detect the occurrence of specific content-based events that match events specified in the user's content aware trigger criteria. At 506, while or after the device 515 inspects the content metadata, the device 515 detects that the user's content aware trigger criteria have been met.

[0059] Option #2 - The content provider generates a content aware behavior notification. At 507 in Figure 5, based on the results of inspecting the content to detect content-based events of interest at 502, the content provider 520 implementing option 2 may generate content aware notifications. These content aware notifications may be piggybacked and sent along with the content that the content provider distributes. The generated content aware notifications may include one or more elements defined in Table 4.

[0060] At 508 of FIG. 5, the device 515 receives multimedia content and a content-aware notification from the content provider 520. In one implementation not specifically shown in FIG. 5 for option 2, a service provider different from the content provider may provide metadata associated with the content providing the content-aware notification. Thus, the content provider and the service provider may be different entities in either the same or different network locations. At 509, the device 515 processes the content-aware notification locally to determine whether the device needs to perform any corresponding content-aware actions. This processing may involve the device comparing content-aware action identifiers present in the received content-aware notification with stored content-aware action identifiers maintained on the device to find a match.

[0061] Looking at options 1 and 2 of FIG. 5, it should be noted that in both cases, at 504 and 508, the device 515 receives metadata to aid in detecting the presence of the trigger criteria. Thus, in option 1, the device 515 receives metadata at 504 to aid in determining whether the trigger criteria are met at steps 505 and 506. In one example, the metadata may be a notification that the trigger criteria are met. As with option 2, the device 515 receives metadata in the form of a notification from a service provider or content provider to aid in subsequent processing at step 509. Thus, the receipt of metadata at steps 504 and 508 is common to both option 1 and option 2 of FIG. 5. Common to both option 1 and option 2, the exemplary device 515 shown in FIG. 5 is a DTV, which may include, but is not limited to, a DTV, a smartphone, a tablet, a laptop, a smartwatch, smart glasses, an AR HMD, a STB with a display attached, and the like.

[0062] Upon detecting that the user's content-aware trigger criteria have been met at 512 in Figure 5, the device 515 may perform one or more content-aware actions, such as, but not limited to, those defined in Table 1. Note that the action at 512 may be performed as a result of performing either option 1 or option 2 in Figure 5.

[0063] [Table 3]

[0064] [Table 4]

[0065] Content notification provider assists in processing content-aware behavior As shown in the procedure defined in FIG. 6, a device may rely on one or more content notification providers that are separate and distinct entities from the content provider and provide content-aware notifications to the device. A device may selectively subscribe to one or more content notification providers that provide notifications that have relevance to multimedia content of interest to a user of the device. These content-aware notifications may be sent to the device separately, rather than piggybacked with the raw content sent from the content provider (i.e., the device receives the content and the content-aware notifications separately and independently of each other). Although the information contained within the content-aware notifications may be used to enable the device to determine if / when to perform a content-aware action, the content-aware notifications may not be synchronized to the content because they originate from entities separate from the content provider. Thus, the device may need to synchronize the content-aware notifications to the content.

[0066] FIG. 6 illustrates a separate content notification provider and content provider configuration. At 601 of FIG. 6, the setup and configuration for content aware operation is accomplished as described above with respect to FIG. 2-FIG. 3. In addition, the device 615 may share the content aware operation information of the user 610 with one or more content notification providers 620. The device 615 may selectively share the user's content aware operation information with only certain content notification providers 620. For example, only those content notification providers that the device determines to provide multimedia content that is of interest to the user and meets the user's content aware operation criteria (i.e., content aware trigger criteria). In one embodiment, the device 615 may share the user's content aware operation information via a subscription request that the device sends to the content notification provider 620. In addition to the user's content aware operation information, this subscription request may also include device callback information (e.g., an address, such as a URI, at which the device 615 can receive notifications from the content notification provider 620).

[0067] 6, 602 and 603, the content provider 630 provides multimedia content to both the device 615 and the content notification provider 620. Alternatively, the notification provider 620 may selectively receive only applicable content based on user default settings and requests, or the device may notify the notification provider 620 to start receiving content only when applicable content is detected.

[0068] In 604 of Figure 6, the content notification provider 620 inspects the multimedia content to detect content-based events of interest. These content-based events of interest may be defined by a current user who configured content-aware behaviors that were forwarded to the content notification provider in 601. Alternatively, the content-based events of interest may be events that the content notification provider 620 feels would be of interest to a prospective user.

[0069] 6, a content notification provider 620 generates content-aware notifications based on the results of inspecting the multimedia content to detect content-based events of interest in 604. These generated content-aware notifications may include one or more elements defined in Table 4. The content-aware notifications are transmitted to the device 615.

[0070] 6, at 606, the device 615 receives a content aware notification from a content notification provider 620. At 607, the device 615 processes the content aware notification locally to determine whether the device 615 needs to perform any corresponding content aware actions. This processing may involve the device 615 comparing content aware action identifiers present in the received content aware notification to stored content aware actions maintained on the device to find a match.

[0071] 6, for any content aware notification that requires the device 615 to perform a content aware action, the device 615 may need to locally synchronize the received content aware notifications to the received multimedia content because they may originate from two independent asynchronous sources (i.e., the content provider 630 and the content notification provider 620). To perform this synchronization, the device 615 may use a content synchronization marker information element included within each content aware notification it receives. Using the content synchronization marker information element, the device can synchronize the notification to the multimedia content, which may involve the device 615 synchronizing the notification and the content based on absolute or relative timestamp information, transmission frame or slot number information.

[0072] 6, once matches for content-aware actions are found and synchronization is performed, the device 615 can extract the content-aware actions that need to be performed from each matching set of content-aware actions. The device 615 can then perform one or more content-aware actions, such as, but not limited to, those described in Table 1.

[0073] System Level Implementation Figure 7 illustrates a representation 700 of several different types of entities in an end-to-end multimedia content distribution and consumption system in which the user-defined content-aware behavior functionality proposed herein may be implemented. Figure 7 illustrates a system-level embodiment of the user-defined content-aware behavior functionality.

[0074] Within a user's personal area network 705, various types of multimedia-centric devices may support the proposed device-centric user-defined content-aware behavior functionality proposed herein. Such devices may include, but are not limited to, DTVs, smartphones, tablets, laptops, smart watches, smart glasses, AR HMDs, etc., not shown in FIG. 7. For example, the devices may support the capabilities to monitor multimedia content, detect user-defined content-aware event criteria, generate content-aware notifications, send content-aware notifications to users, and perform content-aware actions, as described with respect to FIGS. 2-6. These devices may also interact with other entities in the system (e.g., various content providers, various core network nodes) via network connections 760 to support these capabilities.

[0075] Various types of communication core networks 760 may support the user-defined content-aware behavior functionality proposed in the present invention. Such communication core networks may include, but are not limited to, 5G, 4G, satellite, IP, and cable TV networks. Within these networks, different types of network nodes / functions may support the user-defined content-aware behavior functionality. For example, the ability to monitor multimedia content, detect user-defined content-aware event criteria, generate content-aware notifications, send content-aware notifications to devices, and assist devices with performing content-aware actions.

[0076] Various types of content broadcasting core networks may support the user-defined content-aware behavior functionality proposed herein. Such broadcasting core networks 745 may include, but are not limited to, networks based on the ATSC or DVB standards. Within these networks, different types of network nodes / functions may support the user-defined content-aware behavior functionality, such as the ability to monitor multimedia content, detect user-defined content-aware event criteria, generate content-aware notifications, send content-aware notifications to devices, and assist devices with performing content-aware actions.

[0077] Various types of content providers may support the user-defined content-aware behavior functionality proposed herein. Such content providers may include, but are not limited to, those that broadcast content to devices 715, those that stream content to devices 725, and / or those that generate and send content-based notifications to devices 735. These content providers may support user-defined content-aware behavior functionality, such as the ability to monitor multimedia content, detect user-defined content-aware event criteria, generate content-aware notifications, send content-aware notifications to devices, and assist devices with performing content-aware actions.

[0078] Protocol embodiment Figure 8 depicts a system level embodiment of user-defined content aware operations showing various exemplary interfaces. In Figure 8, some potential interfaces within an end-to-end multimedia content distribution and consumption system where user-defined content aware operations functionality may be supported. A user's personal area network 805 may interface with a content broadcast provider 815, a content streaming provider 825, a content notification provider 835. Also shown are interfaces between the user's personal area network 805, the content broadcast network 845, and the communication core network 855. The interface between the content broadcast provider 815 and the content broadcast core network 45 is depicted as is the interface between the content broadcast core network 845 and the communication core network 855.

[0079] Table 5 defines each of the proposed interfaces captured in Figure 8. Each of these interfaces may support the exchange of user-defined content-aware action messages between each of the applicable entities in the system. A collection of these messages may form a protocol used to enable the exchange and execution of user-defined content-aware actions in an end-to-end manner across the entire system.

[0080] [Table 5]

[0081] In one embodiment, the user-defined content-aware action message protocol can be implemented as a client / server messaging protocol, where a user and / or his / her personal device can function in the role of a client and / or a server to exchange user-defined content-aware action request and response messages with other entities (e.g., content providers) in the system. For example, information elements of the user-defined content-aware action request and response protocol messages can be encapsulated and carried within the payload of an existing client / server protocol such as HTTP, COAP, or WebSockets.

[0082] In another embodiment, these information elements may be encapsulated and carried within a lower level protocol, such as TCP or UDP, without using a higher layer protocol, such as HTTP, COAP, or WebSockets.

[0083] In yet another embodiment, user-defined content-aware action messages can be encapsulated and carried within a publish / subscribe messaging protocol. For example, an entity in the system (e.g., a communications core network entity or a content broadcasting core network entity) can support a message broker functionality. This broker functionality can be used by devices to exchange user-defined content-aware action messages with other entities in the system (e.g., content providers). This exchange can be facilitated by each entity subscribing to the message broker to receive messages from other entities. Similarly, each entity can publish messages targeted to other entities to the message broker. Information elements of the user-defined content-aware action message request and response protocol messages can be encapsulated and carried within the payload of an existing publish / subscribe protocol such as MQTT or AMQP.

[0084] In yet another embodiment, the user-defined content-aware action information element may be embedded within a content streaming protocol. For example, but not limited to, streaming protocols such as Real Time Messaging Protocol (RTMP), Real Time Streaming Protocol (RTSP), MPEG-DASH, Apple HTTP Live Streaming (HLS), WebRTC, and SRT. In one type of embodiment, the user-defined content-aware action information element may be embedded within one or more separate virtual channels or streams supported by a streaming protocol (e.g., RTMP). These virtual channels or streams are multiplexed together and transmitted over a single physical transport connection (e.g., TCP). For example, the user-defined content-aware action information element may be transmitted on one channel / stream and the multimedia content may be transmitted on another channel / stream. These two channels / streams may be multiplexed and transmitted together over a single transport connection between entities in the system.

[0085] Various types of user-defined content-aware action message request and response protocol messages may be supported by different entities. These may include, but are not limited to, the message types defined in Table 6.

[0086] [Table 6]

[0087] FIG. 9 illustrates an example flow diagram 900 illustrating aspects of the disclosure. The method of flow diagram 900 may be performed by a viewing device operating in an environment described above. At 905, the viewing device receives a content-aware request for an action to be performed by the viewing device. The request includes trigger criteria that are met for the requested action to be performed. At 910, the viewing device receives digital content, such as digital multimedia content, from a content provider. At 915, a detection of a condition occurs that the trigger criteria for the action to be performed are met. In one embodiment, the condition is detected when a match occurs between the trigger criteria and the received digital content. At 920, the viewing device performs the requested action as a result of detecting that the trigger criteria for performing the action have been met.

[0088] In one embodiment, receiving a request for an action to be performed by the viewing device includes receiving a command from a remote control device to establish at least one trigger criterion related to content not displayed by the viewing device. In this case, the viewing device may be displaying a first digital content, such as a first program being displayed, and the trigger criterion is related to a second digital content, such as a second program not being displayed. In this case, the at least one trigger criterion may be related to the second program not being actively displayed. In one aspect, the content received from the content provider includes digital content and associated metadata. In one aspect, detecting a condition that a trigger criterion for an action to be performed has been met may involve one or more of receiving content and metadata related to the trigger criterion, matching the received metadata to the trigger criterion, and / or receiving a notification from the service provider (or content provider) that the content metadata matches the trigger criterion. In another aspect of the disclosure, the viewing device may perform the requested action by sending a text message, changing a display on the viewing device to display a program associated with the detected trigger criteria, changing a display on the secondary device to display a program associated with the detected trigger criteria, recording multimedia content associated with the detected trigger criteria, and / or displaying a notification on a screen of the viewing device or the secondary device.

[0089] In the example flow 900, a viewing device may receive a request for an action that includes a trigger criterion, the trigger criterion including a specified content-based event occurring within the multimedia content selection. The specified content-based event may include one or more of a start / end of a timeout in a sporting event, a change in the score in a sporting event, a performance by a particular singer, a particular scene in a movie, or the start of a final game show round. Receiving a request for an action that includes the trigger criterion may also include receiving a request where the trigger criterion includes the start or end of a commercial advertisement associated with the multimedia content selection.

[0090] FIG. 10 is a depiction of an exemplary display device architecture 900 for a DTV, laptop computer, desktop computer, etc., configured to perform the method of FIG. 9. The device of FIG. 10 may be a dedicated machine or may be part of a larger machine that performs other tasks. For example, the device of FIG. 10 may be an access point or an access point-enabled device that can support a wireless local area network and a display for users to interact with content. Such machines may be digital televisions, laptops, personal computers, mobile phones, tablets, etc., and are configured to function as interactive devices. Here, for simplicity, the description may follow that of a DTV, but other devices such as those listed above are possible, as will be well understood by those skilled in the art.

[0091] The device 1000 of FIG. 10 includes a transmitter / receiver interface 1002 that provides connectivity to an IP network to receive digital content. The interface 1002 connects to a bus interface 1004 that allows access to an internal bus 1024. Other non-bus implementations are possible as known to those skilled in the art. Residing on the bus 1024 is a storage device 1006 that can be used for any general storage of retrieved or requested data and network management data, parameters, and digital content, as well as executable software instruction programs. The storage device 1006 can also function as a disk or solid state storage device for information collected as displayable digital content. The main executable programs, utilities, and other programs can be under the control of a controller / processor 1008.

[0092] The controller / processor 1008 may be a single processor or multiple processors that perform tasks of digital content data acquisition, user interface control, and resource management. The controller / processor 1008 may execute the methods described in FIG. 9. The control memory 1010 may also provide program instructions and configuration control for the controller / processor 1008. The status indicator 1018 is a user interface that allows a user, system owner, or system administrator to view the status of the device 1000. Such indicators may include displays, LEDs, speakers, etc. A graphical processing unit (GPU) and input / output (I / O) interface 1016 allow the device to connect to a main display screen 1020 that may be used to display digital content on the device 1000. The I / O interface 1016 may also include a hardline interface such as an Ethernet interface for a local area network, or may be an HDMI interface for the main screen 1020. Another option for the I / O interface is an RF or IR interface for remote control of the device 1000. Another option for the I / O interface 1016 is an interactive interface that may include the use of a separate display device (secondary device), a keyboard, a mouse, a light pen, or the like.

[0093] The apparatus 1000 has a wireless network interface 1012 that allows access to and from wireless devices, such as receiving viewer devices, that may be known and registered with the WLAN of the apparatus 1000. Such interface 1012 includes all elements for controlling a wireless network, including the use of wireless network protocols such as IEEE 802.XX, Bluetooth, or other short-range wireless communication protocols for communication between the apparatus 1000 and the receiving viewer devices.

[0094] As can be understood by those skilled in the art, combinations of features of various applications of the present disclosure may be combined unless otherwise specified. Thus, features of various figures may be easily combined. For example, a setup or configuration of the functions described herein may be generated for a DTV using either or both of a personal device, a secondary device, or a hologram simultaneously, such that DTV content-aware operations are performed for the viewer. In that case, the viewer may also invoke a DTV content-aware operation request for a second (or more) viewer to share the DTV viewing experience. Combining features described in the present disclosure is within the scope of the concepts presented herein. Such combinations of concepts and features are considered viable embodiments unless otherwise limited as indicated herein.

[0095] The implementations described herein may be implemented, for example, in a method or process, an apparatus, or a combination of hardware and software. Even when discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms. For example, the implementation may be achieved via a hardware apparatus, a hardware and software apparatus. An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. A method may be implemented in an apparatus, such as a processor, which refers to any processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device.

[0096] In addition, the method may be implemented by instructions executed by a processor, such instructions may be stored on one or more processors or computer-readable media, such as, for example, an integrated circuit, a software carrier, or other storage device, such as, for example, a hard disk, a compact disk ("CD" or "DVD"), a random-access memory ("random-access memory, RAM"), a read-only memory ("read-only memory, ROM"), or any other magnetic, optical, or solid-state medium. The instructions may form an application program tangibly embodied on a computer-readable medium, such as any of the media listed above or media known to those skilled in the art. The instructions so stored are useful for executing the hardware and software elements to perform the steps of the methods described herein. In the present disclosure, the computer-readable medium is useful for instructing either and / or both of the first processor and the second processor to perform the steps of the methods disclosed herein.

Claims

1. 1. A device-implemented method, comprising: receiving a request for an action to be performed by the device, the request including event criteria for detecting an event in content received by the device; receiving metadata associated with the event within the content; detecting the event in the content based on a match between the event criteria and the metadata; and performing the action based on the detection of the event.

2. 2. The method of claim 1 , wherein receiving metadata associated with the event in the content further comprises receiving the content along with the metadata from a content provider in response to sending the request to the content provider.

3. The method of claim 1 , wherein receiving the request for the action to be performed by the device includes receiving a command from a remote control device to establish at least one event criterion related to content not displayed by the device.

4. Detecting the event based on a match between the event criteria and the metadata includes:

2. The method of claim 1, comprising receiving a notification from a service provider that the metadata matches the event criteria, the service provider being a content provider or being separate from the content provider.

5. Performing the action by the device includes: Sending text messages; modifying a display on the device to display a program associated with the detected event; modifying a display on a secondary device to display a program associated with the detected event; recording multimedia content associated with the detected event; displaying a notification on a screen of said device or a secondary device; The method of claim 1 , comprising one or more of:

6. The method of claim 1 , wherein receiving the request for the action to be performed by the device comprises receiving a request in which the event criteria includes a content-based event occurring within a multimedia content selection.

7. 7. The method of claim 6, wherein the content-based events include one or more of the following: a start / end of a timeout in a sporting event, a change in score in a sporting event, a performance by a particular singer, a particular scene in a movie, or the start of a final game show round.

8. A device for receiving and displaying digital content, comprising: a receiver for receiving a request for an action to be performed by the device, the request including event criteria for detecting an event in content received by the device; The receiving unit receives metadata related to the event in the content; a processor, the processor comprising: detecting the event in the content based on a match between the event criteria and the metadata; causing the device to perform the action based on detection of the event.

9. 10. The apparatus of claim 8, wherein the receiver receives the metadata associated with the event in the content by receiving the content together with the metadata from a content provider in response to sending the request to the content provider.

10. The apparatus of claim 8 , wherein the receiver receives commands from a remote control device to establish at least one event criterion related to content not displayed by the apparatus.

11. 10. The apparatus of claim 8, wherein the processor detects the event based on a match between the event criteria and the metadata by detecting a notification from a service provider that content metadata matches the event criteria, the service provider being a content provider or being separate from the content provider.

12. The processor may further include: Sending text messages; modifying a display on the device to display a program associated with the detected event; modifying a display on a secondary device to display a program associated with the detected event; recording multimedia content associated with the detected event; and displaying a notification on a screen of the device or a secondary device.

13. The apparatus of claim 8 , wherein the event criteria comprises a content-based event occurring within a multimedia content selection.

14. 14. The apparatus of claim 13, wherein the content-based events include one or more of the following: a start / end of a timeout in a sporting event, a change in score in a sporting event, a performance by a particular singer, a particular scene in a movie, or the start of a final game show round.

15. 10. The device of claim 8, wherein the device is one of a digital television, a smartphone, a tablet computer, a laptop computer, a desktop computer, a set-top box with an attached display, or an augmented reality head-mounted device.

16. A non-transitory computer-readable storage medium having instructions that, when executed by a processor, receiving a request for an action to be performed by a device, the request including event criteria for detecting an event in content received by the device; receiving metadata associated with the event within the content; detecting the event in the content based on a match between the event criteria and the metadata; performing the action based on the detection of the event; and A non-transitory computer-readable storage medium for executing the method of the present invention.

17. 17. The non-transitory computer-readable storage medium of claim 16, wherein receiving metadata associated with the event in the content further comprises receiving the content along with the metadata from a content provider in response to sending the request to the content provider.

18. 17. The non-transitory computer-readable storage medium of claim 16, wherein receiving the request for the action to be performed by the apparatus includes receiving a command from a remote control device to establish at least one event criterion related to content not displayed by the device.

19. 17. The non-transitory computer-readable storage medium of claim 16, wherein detecting the event based on a match between the event criteria and the metadata includes receiving a notification from a service provider that the metadata matches the event criteria, the service provider being a content provider or being separate from the content provider.

20. Performing the action includes: Sending text messages; modifying a display on the device to display a program associated with the detected event; modifying a display on a secondary device to display a program associated with the detected event; recording multimedia content associated with the detected event; and displaying a notification on a screen of the device or a secondary device.

Images