Context extraction from video for enhanced television shopping experience using voice interface
Patent Information
- Application Number
- PCT/IB2024/054456
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-08
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-14
Smart Images

Figure IB2024054456_14112024_PF_FP_ABST
Abstract
Description
CONTEXT EXTRACTION FROM VIDEO FOR ENHANCED TELEVISION SHOPPINGEXPERIENCE USING VOICE INTERFACEField of the Disclosure
[0001] The present disclosure relates generally to a method, system, and computer program product for context extraction from media based on user voice input, and more particularly to identifying objects in a piece of media content in response to a user voice input.Background of the Disclosure
[0002] Voice User Interfaces (VUIs) allow users to interact with devices using voice commands or speech such that users need not use hand gestures or control devices, e.g., remotes, keyboards, etc. In typical VUI enabled devices, users interact with a Voice Assistant (VA) which is built into the device as software or hardware. While speaking with a VA, it is important that the VA knows the context of the voice commands issued by the user. For example, a simple question like “what is the time” does not require any further context because the VA knows what time zone it is in or has been set to, but a request like “book my tickets” requires more context for the VA to understand what is meant. In response to a voice command lacking sufficient context, a VA could ask additional questions to understand the context, or it may determine the context using previous conversations and ask the user to confirm. For example, a conversation with sufficient initial context might be: User: “VA, I am planning to visit my friend in Irving this Friday;” VA: “Do you want to take a flight?;” User: “Yes, book my tickets.” In such a conversation, when user says “Yes, book my tickets,” the VA hassufficient context like the user is travelling to Irving (place) on Friday (date) by flight (mode of transport) to book an airline ticket. Thus, context is very important for a VA to function correctly.
[0003] In a particular environment, a VUI may be used to control a set-top box (STB). For example, a user may mute a STB using VUI by issuing a voice command to a VUI enabled device in communication with the STB, e.g., a smart media device, a mobile phone, a Bluetooth remote connected to the STB, etc., or directly to a VUI enabled STB. The VUI enabled device may communicate directly with the STB or it may communicate with the STB via a cloud service, e.g., a voice cloud. The voice cloud may be a third- party cloud service associated with the VUI enabled device. The voice cloud will interpret the voice command received by the Ul enabled device and convert it into an appropriate computer-readable command. The voice cloud may forward the converted command to a device cloud, which determines a target device, e.g., the STB, where the command is to be executed. The device cloud transmits the command to the STB and the STB is muted upon receiving and executing the command. In another environment, a VUI enabled device may be used in online shopping. For example, a VUI enabled device may allow a user to add items to a virtual shopping cart using a voice command. For example, a user may want to purchase a particular product and may issue a voice command to the VUI enabled device such as “VA, add one liter of branded laundry detergent to my shopping cart.” In this example, as above, the VA requires sufficient information in order to effectuate the voice command. If a user doesn’t provide sufficient information, the VA will engage in a conversation with the user to understand the context to select the right product or it might ask user to confirm the product selection inan associated application or website. Therefore, in current uses of VUI enabled devices to perform online shopping experiences, the VUI enabled devices must receive a very detailed voice command in order for there to be sufficient context for the VUI enabled device to understand the voice command without further user input. The required level of detail and context needed in current VUI enabled shopping experiences may be prohibitive to some users. For example, some users may determine that direct interaction with a device may be quicker than getting a VUI enabled device to understand what the user wants. Further, longer conversations with a VA can result in further confusion depending on the VA’s ability to understand certain users, e.g., accents, etc. Thus, there is a need for a technical solution for enabling a VUI-enabled device to determine the context of a voice command without additional user input.Summary of the Disclosure
[0004] The present disclosure provides a description of exemplary methods, systems, and computer program products for context extraction from media based on user voice input in accordance with exemplary embodiments.
[0005] A method for context extraction from media based on user voice input, the method comprising: receiving, by computer premises equipment (CPE ), a user voice input including a reference to an object in a piece of media content and an object request, the piece of media content being displayed on a display device connected to the CPE; capturing, by the CPE, a segment of the piece of media content upon receipt of the user voice input; analyzing, by the CPE, the captured segment for the object referenced in the user voice input; identifying, by the CPE, the object in the segment ofthe media content; and transmitting, by the CPE, an instruction to an external computing device, the instruction based on the object request.
[0006] A system for context extraction from media based on user voice input, the system comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the instructions comprising: instructions to receive a user voice input including a reference to an object in a piece of media content and an object request, the piece of media content being displayed on a display device; instructions to capture a segment of the piece of media content upon receipt of the user voice input; instructions to analyze the captured segment for the object referenced in the user voice input; instructions to identify the object in the segment of the media content; and instructions to transmit an instruction to an external computing device, the instruction based on the object request.
[0007] A computer program product for context extraction from media based on user voice input, the computer program product comprising: a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method, comprising: receiving, by consumer premises equipment (CPE), a user voice input including a reference to an object in a piece of media content and an object request, the piece of media content being displayed on a display device connected to the CPE; capturing, by the CPE, a segment of the piece of media content upon receipt of the user voice input; analyzing, by the CPE, the captured segment for the object referenced in the user voiceinput; identifying, by the CPE, the object in the segment of the media content; and transmitting, by the CPE, an instruction to an external computing device, the instruction based on the object request.Brief Description of the Drawings
[0008] The scope of the present disclosure is best understood from the following detailed description of exemplary embodiments when read in conjunction with the accompanying drawings. Included in the drawings are the following figures:
[0009] FIG. 1 is a block diagram illustrating a high-level system architecture for context extraction from media based on user voice input in accordance with exemplary embodiments;
[0010] FIG. 2 is a block diagram illustrating the consumer premises equipment of FIG. 1 for context extraction from media based on user voice input in accordance with exemplary embodiments;
[0011] FIGS. 3A-3B combined is a flow diagram illustrating a process for context extraction from media based on user voice input in accordance with exemplary embodiments;
[0012] FIG. 4 is a flow diagram illustrating an exemplary method for context extraction from media based on user voice input in accordance with exemplary embodiments;
[0013] FIGS. 5A-5C are diagrams illustrating content segments of a piece of a media content in accordance with exemplary embodiments; and
[0014] FIG. 6 is a block diagram illustrating a computer system architecture in accordance with exemplary embodiments.
[0015] Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description of exemplary embodiments are intended for illustration purposes only and are, therefore, not intended to necessarily limit the scope of the disclosureDetailed Description of the Preferred Embodiments and Methods
[0016] The present disclosure provides a novel technical solution for context extraction from media based on user voice input. Particularly, exemplary embodiments of the methods, systems, and computer program products provided for herein enable a VUI- enabled device to determine the context of a voice command without additional user input by analyzing a piece of media content captured by a consumer premises equipment (CPE) device, e.g., cable / satellite set-top boxes (STB) and Digital Subscriber Line (DSL) routers, digital assistants, and any other device (e.g., terminals and associated equipment) located at a subscriber's premises and connected with a carrier's telecommunication circuit or multichannel video service (MSO) or streaming service for example on the consumer’s side of a demarcation point. For example, if a user is watching a commercial for a particular product and wants to purchase that product, the user may issue a voice command to a VUI enabled device (for example, a VI remote controller, whether a dedicated device, a smart phone, personal assistant, etc. that might process the voice information on site or remotely) such as “add that to my cart.” In this example, the VUI enabled device locally or remotely processes the command and determines a secondary device is to be called upon (e.g., a CPE currently in use by the user), and sends the command to that secondary device. The CPE captures a segmentof the media content being displayed to the user and locally or remotely analyzes that segment to determine what “that” refers to in the voice command. In current VUI enabled devices, user voice commands must provide sufficient details and context in order for the CPE to understand the user voice commands. The required level of detail and context needed in current VUI enabled voice commands may be prohibitive to some users. Exemplary embodiments of the methods, systems, and computer program products provided for herein provide a technical solution for providing content to user voice commands received by a VUI enabled device using a CPE. Thus, the methods, systems, and computer program products provided for herein provide a novel way for context extraction from media based on user voice input received by a VUI enabled device.System for Context Extraction from Media Based on User Voice Input
[0017] FIG. 1a illustrates an exemplary system 100 for context extraction from media based on user voice input. The system 100 includes a user 102, user devices 104, a remote control device 106, a consumer premises equipment (CPE) 108, a voice cloud 120, a device cloud 130, an external computing device 140, multiple system operator (MSO) 150, and servers 152.
[0018] In embodiments, the CPE 108 can provide, for example, voice, video, data, and / or loT services to one or more of the user devices 104. The CPE 108 may communicate with the one or more of user devices 104 over a local network 110. The local network 110 may be, for example, but not limited to, a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), a mobilecommunication network, a satellite network, the Internet, fiber optic, coaxial cable, infrared, radio frequency (RF), or any combination thereof. The CPE 108 may communicate with an upstream wide area network (WAN) through a connection 160 to the one or more servers 152, for example, of the multiple system operator (MSO) 150. The one or more servers 152 can provide high-bandwidth data transfer, for example, cable television and broadband Internet access via, for example, the connection 160. The connection 160 may be, but is not limited to, a wired connection, a wireless network (e.g., WiFi), a mobile communication network, a satellite network, the Internet, fiber optic, coaxial cable, infrared, radio frequency (RF), or any combination thereof. Other suitable network types and configurations for the local network 110 and the connection 160 will be apparent to persons having skill in the relevant art. In general, the local area network 110 and / or the connection 160 can be any combinations of connections and protocols that will support communications between the CPE 108 and the user devices 104 and the MSO 150.
[0019] In accordance with an exemplary embodiment, the CPE 108 and the one or more of user devices 104 can be configured to connect via a wireless network, for example, a wireless network utilizing an IEEE 802.11 specification. The one or more of user devices 104 include, but are not limited to, a smart phone, a television, a display monitor, a computer, a mobile device, a tablet, a router, a home security system, or any other electronic device or computing system capable of storing, compiling, and organizing audio, visual, or textual data and receiving and sending that data to and from other computing devices, such as the remote control device 106, the CPE 108, and the voice cloud 120. Further, it can be appreciated that the user devices 104 may includeone or more computing devices. In an exemplary embodiment, one or more of the user devices 104 is a voice-enabled user device such as, but not limited to, a voice-enabled digital assistant device (e.g., an Amazon® Alexa device, an Apple® Sih® device, a Google Home® assistant device, or a Microsoft® Cortana® device, etc.). As illustrated, the user devices 104 may include, but are not limited to, a smartphone 104a, a voice- enabled digital assistant device 104b, and a display 104c. The CPE 108 may provide access to an external network, such as the Internet, for any devices connected thereto via the local network 110. In accordance with an exemplary embodiment, the CPE 108 may be a smart media device, a gateway device, an access point, a modem, a wireless router including an embedded modem, a wireless network extender or any other device operable to deliver, for example, data and / or video services from the one or more servers 152 of the MSO 150 and / or a wide area network (WAN) to one or more of the user devices 104. The one or more of user devices 104, the CPE 108, and the local network 110 may be associated with a user 102. While only a single user 102 is illustrated, it can be appreciated that any number of users may be associated with the user devices 104, the CPE 108, and the local network 110. The user 102 may interact with the CPE 108 via the remote-control device 106. The remote-control device 106 can be any known remote-control unit having various buttons that control various functionalities relating to the device being controlled, e.g., the CPE 108. For example, buttons on the remote-control device 106 may take a user to a guide screen, change channels up or down, or access other applications associated with the CPE 108. The remote-control device 106 is programmed to recognize the actuation of keys or control surfaces associated with the various functionalities. Typically, the signals generated bythe remote-control device 106 are transmitted over a wireless RF link or an IR optical link. In embodiments, the remote-control device 106 may include a microphone and the various functionalities of the remote-control device 106 may be voice activated, e.g., via voice commands from the user 102.
[0020] In accordance with an exemplary embodiment, the CPE 108 may communicate with the one or more servers 152 over a wired or a wireless connection. A wireless connection between the one or more servers 152 of the MSO 150 and the CPE 108 may be established through a protected setup sequence (for example, Wi-Fi protected setup (WPS)). The protected setup sequence may include the steps of scanning multiple wireless channels for an available access point, exchanging one or more messages between a station and access point, exchanging key messages (for example, pre-shared key (PSK)) between the station and access point, and installing a key (for example, PSK) at the station.
[0021] The system 100 may include a voice cloud 120. The voice cloud 120 may include one or more servers (not pictured) operated by a cloud service operator. The cloud service operator may be the MSO 150 or any other third-party cloud service operator associated with one or more of the user devices 104 and / or the CPE 108, e.g., Amazon, Apple, Google, or Microsoft, etc. The voice cloud 120 may be associated with one or more of the user devices 104. For example, the voice cloud 120 may be operated by the MSO 150 and the user device 104b may be a voice-enabled digital assistant device manufactured by the MSO 150. Alternatively, the voice cloud 120 may be operated by a third-party device cloud operator and the user device 104b may be a voice-enabled digital assistant device manufactured by the third-party. The one or more servers of thevoice cloud 120 may be any type of electronic device or computing system configured to store, compile, and organize audio, visual, or textual data and receiving and sending that data to and from other computing devices, such as the one or more user devices 104, the CPE 108, and / or the device cloud 130. The voice cloud 120 may include one or more user profiles associated with the user device 104. For example, the voice cloud 120 may include a user profile for the user 102 and the user profile may identify one or more user devices 104 associated with the user 102. The user profiles stored by the voice cloud 120 may identify other devices associated with the user 102 such as, but not limited to, the CPE 108, and any cloud services associated with those other devices, e.g., the device cloud 130. The user profiles by the voice cloud 120 may identify accounts with the external computing device 140 associated with the user 102. In an exemplary embodiment, the voice cloud 120 receives a voice command from one or more of the user devices 104, interprets the voice command, determines a target device(s) of the voice command (e.g., based on the user profile), converts the voice command into a computer executable command, and transmits the converted command to the device cloud 130 and / or the CPE 108. While only a single voice cloud 120 is illustrated, it can be appreciated that any number of voice clouds may be a part of the system 100. Further, in embodiments, the voice cloud 120 may be optional and the user device 104 may store the user profiles and perform the functions of the voice cloud 120 as described above. For example, the user device 104b may interpret the voice command, determine a target device of the voice command, convert the voice command into a computer executable command, and transmit the converted command to the device cloud 130 and / or the CPE 108.
[0022] The system 100 may include a device cloud 130. The device cloud 130 may include one or more servers (not pictured) operated by a device service operator. The device service operator may be the MSO 150 or any other third-party cloud service operator associated with the CPE 108. The device cloud 130 is associated with one or more CPE devices, e.g., all CPE devices issued or manufactured by the MSO 150. The one or more servers of the device cloud 130 may be any type of electronic device or computing system configured to store, compile, and organize audio, visual, or textual data and receiving and sending that data to and from other computing devices, such as the one or more user devices 104, the CPE 108, the voice cloud 120, and / or the external computing device 140. The device cloud 130 may include one or more user profiles associated with the CPE 108. For example, the device cloud 130 may include a user profile for the user 102 and the user profile may identify one or more CPEs associated with the user 102, e.g., the CPE 108. The user profiles stored by the device cloud 130 may identify other devices associated with the user 102 such as, but not limited to, the user devices 104, and any cloud services associated with those other devices, e.g., the voice cloud 120. The user profiles by the device cloud 130 may identify accounts with the external computing device 140 associated with the user 102. In an exemplary embodiment, the device cloud receives the converted voice command from the voice cloud 120, or a user device 104, determines a target device from the converted command, e.g., the CPE 108, where the converted command is to be executed, and transmits the converted command to the target device, e.g., the CPE 108. Further, in embodiments, the device cloud 130 may be optional and the CPE 108 may store the user profiles and perform the functions of the device cloud 130 asdescribed above. For example, the CPE 108 may receive the converted voice command from the voice cloud 120, or a user device 104 and execute the command.
[0023] The external computing device 140 may be a server, a desktop computer, a notebook, a laptop computer, a tablet computer, a handheld device, a smart-phone, a thin client, or any other electronic device or computing system capable of storing, compiling, and organizing audio, visual, or textual data and receiving and sending that data to and from other computing devices, such as the CPE 108 and the device cloud 130. Further, it can be appreciated that the external computing device 140 may include one or more computing devices. In embodiments, the external computing device 140 may be a third-party web-site operator or service provider capable of executing one or more instructions received from the CPE 108 and / or the device cloud 130 or the external computing device 140 may be a third-party web-site operator that maintains an account associated with the user 102. For example, the external computing device 140 may operate a retail website and the external computing device 140 may receive and execute an instruction from the CPE 108 and / or the device cloud 130 to add one or more products to a shopping cart, e.g., a shopping cart associated with the user 102. In another example, the external computing device 140 may be a search engine operator and the external computing device 140 may receive and execute an instruction from the CPE 108 and / or the device cloud 130 to conduct a search on a particular object. While only a single external computing device 140 is illustrated, it can be appreciated that any number of external computing devices may be a part of the system 100.Consumer Premises Equipment
[0024] FIG. 2 illustrates an embodiment of an example CPE 108 in the system 100. It will be apparent to persons having skill in the relevant art that the embodiment of the CPE 108 illustrated in FIG. 2 is provided as illustration only and may not be exhaustive to all possible configurations of the CPE 108 suitable for performing the functions as discussed herein. For example, the computer system 600 illustrated in FIG. 6 and discussed in more detail below may be a suitable configuration of the CPE 108.
[0025] The CPE 108 may include a receiving device 202. The receiving device 202 may be configured to receive data over one or more networks via one or more network protocols. In some instances, the receiving device 202 may be configured to receive data from the user devices 104, the remote-control device 106, the voice cloud 120, the device cloud 130, the external computing device 140, the MSO 150, and other systems and entities via one or more communication methods, such as radio frequency, local area networks, wireless area networks, cellular communication networks, Bluetooth, the Internet, etc. In some embodiments, the receiving device 202 may be comprised of multiple devices, such as different receiving devices for receiving data over different networks, such as a first receiving device for receiving data over a local area network and a second receiving device for receiving data via the Internet. The receiving device 202 may receive electronically transmitted data signals, where data may be superimposed or otherwise encoded on the data signal and decoded, parsed, read, or otherwise obtained via receipt of the data signal by the receiving device 202. In some instances, the receiving device 202 may include a parsing module for parsing the received data signal to obtain the data superimposed thereon. For example, thereceiving device 202 may include a parser program configured to receive and transform the received data signal into usable input for the functions performed by the CPE 108 to carry out the methods and systems described herein.
[0026] The receiving device 202 may be configured to receive data signals electronically transmitted by the user devices 104, the remote control device 106, the voice cloud 120, and the device cloud 130 that may be superimposed or otherwise encoded with commands received from the user 102. For example, the receiving device 202 may receive a user voice command from a user device 104, the remote control device 106, the voice cloud 120, or the device cloud 130. The user voice command may be related to content, e.g., an advertisement, being viewed via the CPE 108 such as on a display device, e.g., the user device 104c, connected to the CPE 108. The receiving device 202 may also receive data signals electronically transmitted by the user devices 104, the remote-control device 106, the voice cloud 120, and the device cloud 130 for controlling one or more interfaces, one or more functionalities, and / or one or more displays, etc. associated with the CPE 108. For example, the receiving device 202 may receive a data signal for executing a channel change, a trickplay operation, a volume control, a display setting change, etc.
[0027] The CPE 108 may also include a communication module 204. The communication module 204 may be configured to transmit data between modules, engines, databases, memories, and other components of the CPE 108 for use in performing the functions discussed herein. The communication module 204 may be comprised of one or more communication types and utilize various communication methods for communications within a computing device. For example, thecommunication module 204 may be comprised of a bus, contact pin connectors, wires, etc. In some embodiments, the communication module 204 may also be configured to communicate between internal components of the CPE 108 and external components of the CPE 108, such as externally connected databases, display devices, input devices, etc. The CPE 108 may also include a processing device. The processing device may be configured to perform the functions of the CPE 108 discussed herein as will be apparent to persons having skill in the relevant art. In some embodiments, the processing device may include and / or be comprised of a plurality of engines and / or modules specially configured to perform one or more functions of the processing device, such as the querying module 214, the analysis module 216, and the generation module 216, etc. As used herein, the term “module” may be software or hardware particularly programmed to receive an input, perform one or more processes using the input, and provides an output. The input, output, and processes performed by various modules will be apparent to one skilled in the art based upon the present disclosure.
[0028] The processing server 102 may include a content database 206. The content database 206 may be configured to store a plurality of content segments 208 using a suitable data storage format and schema. The content database 206 may be a relational database that utilizes structured query language for the storage, identification, modifying, updating, accessing, etc. of structured data sets stored therein. Each content segment 208 may be, but is not limited to, a segment of a piece of media content captured by the CPE 108 in response to receiving a user voice input or command from the user device 104, the voice cloud 120, or the device cloud 130. For example, with reference to FIG. 5A, a piece of media content 502 may be presented to the user 102on a display, e.g., the user device 104c, via the CPE 108. The piece of media content 502 may include, but is not limited to, content segment 504, advertisement segment 506, advertisement segment 508, and content segment 510. The content segment 208 may be a screenshot of the piece of media content 502 at time X, e.g., during advertisement segment 508. Referring to FIG. 5B, the content segment 208 may a predetermined length of video data associated with the piece of media content 502 based on a time X that a user voice input is received by the CPE 108. Referring to FIG. 5C, the content segment 208 may be a cache of all video data associated with the piece of media content 502 being displayed via the CPE 108 prior to a time X that the user voice input is received by the CPE 108.
[0029] The CPE 108 may also include a memory 212. The memory 212 may be configured to store data for use by the CPE 108 in performing the functions discussed herein, such as public and private keys, symmetric keys, etc. The memory 212 may be configured to store data using suitable data formatting methods and schema and may be any suitable type of memory, such as read-only memory, random access memory, etc. The memory 212 may include, for example, encryption keys and algorithms, communication protocols and standards, data formatting standards and protocols, program code for modules and application programs of the processing device, and other data that may be suitable for use by the CPE 108 in the performance of the functions disclosed herein as will be apparent to persons having skill in the relevant art. In some embodiments, the memory 212 may be comprised of or may otherwise include a relational database that utilizes structured query language for the storage, identification, modifying, updating, accessing, etc. of structured data sets stored therein.The memory 212 may be configured to store, for example, the content database 206 and / or the content segments 208.
[0030] The CPE 108 may include a querying module 214. The querying module 214 may be configured to execute queries on databases to identify information. The querying module 214 may receive one or more data values or query strings, and may execute a query string based thereon on an indicated database, such as the content database 206 of the CPE 108 to identify information stored therein, e.g., the content segments 208. The querying module 214 may then output the identified information, e.g. a content segment 208, to an appropriate engine or module of the CPE 108 as necessary. The querying module 214 may, for example, execute a query on the content database 206 to identify a content segment 208 captured by the CPE 108.
[0031] The CPE may include an analysis module 216. The analysis module 216 may be configured to analyze or otherwise process data for use by the CPE 108 in performing the functions discussed herein. The analysis module 216 may be configured to receive instructions to capture a segment of a piece of media content being output by the CPE 108, analyze that segment for one or more objects, analyze instructions for an external computing system identification, and output a result of the analysis to one or more modules of the CPE 108. For example, the analysis module may receive a computer executable command, e.g., from the receiving device 202 via the communication module 204, from a user device 104, the voice cloud 120, or the device cloud 130 to capture a segment of a piece of media content being output by the CPE 108. The analysis module 216 may capture the content segment 208, which may be, but is not limited to, a screenshot of a piece of media content being displayed via the CPE 108, apre-determined length of video data associated with a piece of media content being displayed via the CPE 108 based on a time the computer executable command based on the user voice input is received by the CPE 108, and / or a cache of all video data associated with a piece of media content being displayed via the CPE 108 prior to a time the computer executable command based on the user voice input is received by the CPE 108. The analysis module 216 may analyze the captured segment 208 to identify an object referred to in the computer executable command based on the user voice input. The analysis module 216 may identify the object referred to in the computer executable command based on the user voice input by analyzing, for example, data, metadata, or a quick response (QR) code contained in or associated with the content segment 208. The analysis module 216 may extract the metadata or the QR code from the captured segment 208, e.g., from an SI table or DASH manifest, etc., in order to identify the object. For example, the content segment 208 may be a screenshot of an advertisement being displayed on the user device 104c connected to the CPE 108 and the analysis module 216 may extract an image of a particular product, metadata related to the advertisement that identifies a particular product, or a QR code that identifies the particular product, etc. The analysis module 216 may identify the object referred to in the computer executable command based on the user voice input using natural language processing, image processing, machine learning, or any other known and suitable method to analyze the data of the captured segment 208 to identify the object referred to in the user voice command. Further, the analysis module 216 may analyze the received computer executable command based on the user voice input to identify the external computing device 140. For example, the user voice input may be “add thatto my cart,” and the analysis module 216 determines that the “cart” refers to a virtual shopping cart associated the external computing device 140, which may be a website for the identified object. In embodiments, the external computing device 140 may be a third-party computing system with an account associated with the user 102. The account associated with the user 102 on the external computing device 140 may be identified in or otherwise linked to the user profile stored on the user device 104, the voice cloud 120, and / or the device cloud 130, and may be included in the computer executable command based on the user voice input. In embodiments, the account associated with the user 102 on the external computing device 140 may be an account associated with the CPE 108.
[0032] The CPE 108 may also include a generation module 218. The generation module 218 may be configured to generate data for use by the CPE 108 in performing the functions discussed herein. The generation module 218 may receive instructions as input, may generate data based on the instructions, and may output the generated data to one or more modules of the CPE 108. In embodiment, the generation module 218 may be configured to generate an instruction to the external computing device 140 based on the user voice input. For example, the user voice input may be “add that to my cart,” and once the CPE 108 has identified the object, e.g., “that,” and an external computing device 140 referred to in the user voice input as discussed herein, the CPE 108 may generate an instruction to add the identified object to an online cart associated with the user 102. In another instance, the user voice input may be “who is that,” and once the CPE 108 has identified the object, e.g., “that,” and an external computingdevice 140, e.g., a third-party search engine, the CPE 108 may generate an instruction to search for the identified object.
[0033] The CPE 108 may also include a transmitting device 220. The transmitting device 220 may be configured to transmit data over one or more networks via one or more network protocols. In some instances, the transmitting device 220 may be configured to transmit data to the user device 104, the remote-control device 106, the voice cloud 120, the device cloud 130, the external computing device 140, the MSO 150, and other entities via one or more communication methods, local area networks, wireless area networks, cellular communication, Bluetooth, radio frequency, the Internet, etc. In some embodiments, the transmitting device 220 may be comprised of multiple devices, such as different transmitting devices for transmitting data over different networks, such as a first transmitting device for transmitting data over a local area network and a second transmitting device for transmitting data via the Internet. The transmitting device 220 may electronically transmit data signals that have data superimposed that may be parsed by a receiving computing device. In some instances, the transmitting device 220 may include one or more modules for superimposing, encoding, or otherwise formatting data into data signals suitable for transmission.
[0034] The transmitting device 220 may be configured to electronically transmit data signals to the user device 104, the remote-control device 106, the voice cloud 120, the device cloud 130, the external computing device 140, the MSO 150, which may be superimposed or otherwise encoded with instructions related to a computer executable command based on a user voice input received by the CPE 108. For example, thetransmitting device 220 may transmit an instruction generated by the generation module218 to the external computing device 140.Process for Context Extraction from Media Based on User Voice Input
[0035] FIGS. 3A and 3B illustrate a process for context extraction from media based on user voice input in accordance with exemplary embodiments.
[0036] In the block 302 the user 102 transmits a user voice input to a user device 104. The user voice input may include at least a reference to an object in a piece of media content and an object request. For example, the user 102 may transmit a user voice input to a voice-enabled user device 104b, such as, but not limited to, a smart user device having a voice user interface (VUI). The piece of media content may be, but is not limited to, a live television program, a live movie, a live advertisement, a recorded television program, a recorded movie, or a recorded advertisement, etc. The user voice input may be an object request for an object in a piece of media content being presented to the user 102 on a display, e.g., the user device 104c, via the CPE 108.While an “object” is referred to throughout this disclosure, it can be appreciated that the the term “object” may refer to a reference to a person, a place, a thing, a service, or anything else being presented to the user 102 in the piece of media content and the object request may be any request relating to that reference. For example, the user 102 may be viewing a television program on a display, e.g., the user device 104c, and during an advertisement break in the television program the user 102 may see an product the user 102 would like to purchase or get more information about. In embodiments, the reference to the object and / or the object request of the user voiceinput may be ambiguous. Continuing with the previous example, the user 102 may transmit a user voice input to the user device 104 such as, but not limited to, “add that to my cart,” “I want to know more about that,” “add fifty of those to my cart,” “how much does that cost?,” “who / what is that?,” etc. Thus, the object is not directly referenced in the user voice input, e.g., the user voice input only identifies “that” or “those.” It can be appreciated that any ambiguous reference may be used in the user voice input such as, but not limited to, “he / him,” “she / her,” “they / them,” “it,” etc.
[0037] In embodiments, the user device 104 and the CPE 108 may be the same device. For example, the CPE 108 may include a microphone capable of receiving user voice input and / or may be connected to a remote-control device 106 that is capable of receiving user voice input. In embodiments where the user device 104 and the CPE 108 are the same device, the user voice input is transmitted to the CPE 108 either directly or via the remote-control device 106 and the process may proceed to directly block 308 from block 302.
[0038] In the block 304, the user voice input is received by the user device 104. The user device 104 processes the user voice input to determine one or more target devices, e.g., the CPE 108 and / or the external computing device 140 in block 305. For example, the user device 104 may identify a profile associated with the user 102 and the profile may identify one or more other devices associated with the user 102, e.g., the CPE 108 and / or the external computing device 140. The process of block 305 may also be performed by with the voice cloud 120 and / or the device cloud 130.
[0039] In block 306, the user device 104 transmits the user voice input to the CPE 108. The user device 104 may be communicate directly with the CPE 108, e.g., via the localnetwork 110, or the user device 104 may communicate with the CPE 108 via one or more cloud services, e.g., the voice cloud 120 and / or the device cloud 130. In embodiments where the user device 104 communicates directly with the CPE 108, the user device transmits the user voice input to the CPE in block 306. In embodiments where the user device 104 communicates with the CPE 108 via one or more cloud services, the user device 104 transmits the user voice input to the voice cloud 120 in block 306 and the process may proceed to step 322. In block 322, the voice cloud 120 receives the user voice input. The voice cloud 120 interprets the user voice input and converts the user voice input into an appropriate computer command that can be understood by a computing device, e.g., the CPE 108. The voice cloud may interpret and convert the user voice input using natural language processing or any other known process as will be appreciated by those of ordinary skill in the art of language processing. The voice cloud 120 may be communicate directly with the CPE 108 or the voice cloud 120 may communicate with the CPE 108 via one or more other cloud services, e.g., the device cloud 130. The voice cloud transmits the processed user voice input to the CPE 108 or the device cloud 130 in block 324. In block 326, the device cloud 130 receives the processed user voice input and determines the target device, e.g., the CPE 108, where the user voice input must be executed. For example, the device cloud may include a user profile for the user 102 that identifies computing devices associated with the user 108, e.g., the CPE 108. In block 328, the device cloud 130 transmits the user voice input to the CPE 108. In embodiments where the user profile stored in the user device 104, the voice cloud 120 and / or the device cloud 130 includes more than one device associated with the user 102, the user device 104, thevoice cloud 120 and / or the device cloud 130 may determine which device is currently in use by the user 102.
[0040] Returning now to block 308, the CPE 108 receives the user voice input including the reference to the object in the piece of media content and the object request. As discussed above, the user voice input may be received from the user device 104, the remote-control device 106, the voice cloud 120, or the device cloud 130.
[0041] In the block 310, the CPE 108 captures a segment of the piece of media content upon receipt of the user voice input. The segment of the piece of media content may be, but is not limited to, a screenshot of the piece of media content at the time the user voice input is received, a pre-determined length of video data associated with the piece of media content based on the time the user voice input is received, or a cache of all video data associated with the piece of media content prior to the time the user voice input is received.
[0042] In the block 312 the CPE 108 analyzes the captured segment for the object referenced in the user voice input. The CPE 108 may analyze the captured segment to derive a context and identify the object referenced in the user voice input. Continuing with the example above, if the user voice input is “add that to my cart,” the CPE will analyze the captured segment to determine what “that” is referring to in the captured segment. For example, the user 102 may be watching an advertisement for a particular household product, e.g., a particular laundry detergent product, and transmit the user voice input “add that to my cart.” The CPE 108 analyzes the captured segment to determine “that” refers to the particular product of laundry detergent product. The CPE 108 may, for example, analyze the captured segment by extracting data from thecaptured segment using text extraction techniques, extracting metadata from the captured segment, or extracting a quick response (QR) code from the captured segment.
[0043] In the block 314, the CPE 108 identifies the object in the segment of the media content. The CPE 108 may identify the object based on the extracted data, metadata, or QR code, etc. For example, the CPE 108 may identify the name of a product, e.g., a particular laundry detergent product.
[0044] In the block 316, the CPE 108 generates an instruction based on the identified object and the object request of the user voice input. For example, if the user voice input is “add that to my cart” and the CPE 108 identifies “that” as being a particular laundry detergent product, the CPE 108 may generate an instruction to the “cart” of the user voice input, e.g., the external computing device 140, to add the particular laundry detergent product to a cart associated with the user 102. The “cart” or external computing device 140 may be, for example, an e-commerce server, and the instruction is a request to add the quantity of the object, e.g., the particular laundry detergent product, to an e-commerce shopping cart associated with an e-commerce account of the user 102.
[0045] In the block 318 the CPE 108 transmits the instruction to the external computing device 140, which receives the instructions in block 320.Exemplary Method for Context Extraction from Media Based on User Voice Input
[0046] FIG. 4 illustrates a flow chart of an exemplary method 400 for context extraction from media based on user voice input in accordance with exemplary embodiments.
[0047] In block 402, a user voice input is received by a receiver (e.g., the receiving device 202) of the CPE 108. The user voice input may include at least a reference to an object in a piece of media content being displayed on a display device connected to the CPE 108, and an object request. The user voice input may be received by the CPE 108 from the user 102, a user device 104, the remote control device 106, the voice cloud 120, or the device 130. For example, the object request of the user voice input may be a request to purchase a quantity of the object, e.g., “add that to my cart.” The piece of media content may be, but is not limited to, a live television program, a live movie, a live advertisement, a recorded television program, a recorded movie, and a recorded advertisement.
[0048] In block 404, the CPE 108 captures a segment of the piece of media content upon receipt of the user voice input. The segment of the piece of media content may be, but is not limited to, a screenshot of the piece of media content at the time the user voice input is received, a pre-determined length of video data associated with the piece of media content based on a time the user voice input is received, or a cache of all video data associated with the piece of media content prior to a time the user voice input is received, etc.
[0049] In block 406, the CPE 108 analyzes (e.g., the analysis module 216) the captured segment for the object referenced in the user voice input. The CPE 108 (e.g., the analysis module 216) may analyze the captured segment by extracting, metadata, or a QR code from the captured segment.
[0050] In block 408, the CPE 108 identifies (e.g., via the analysis module 216) the object in the captured segment of the media content. For example, the CPE 108 may identify a product name of the object.
[0051] In block 410, a transmitter (e.g., the transmitting device 220) of the CPE 108 transmits an instruction to the external computing device 140. The instruction based on the object request.
[0052] the external computing device is an e-commerce server, and the instruction is a request to add the quantity of the object to an e-commerce shopping cart associated with an e-commerce account of the userComputer System Architecture
[0053] FIG. 6 illustrates a computer system 600 in which embodiments of the present disclosure, or portions thereof, may be implemented as computer-readable code. For example, the user devices 104, the remote-control 106, the CPE 108, the voice cloud 120, the device cloud 130, and the external computing device 140 of FIG. 1a, may be implemented in the computer system 600 using hardware, software executed on hardware, firmware, non-transitory computer readable media having instructions stored thereon, or a combination thereof and may be implemented in one or more computer systems or other processing systems. Hardware, software, or any combination thereof may embody modules and components used to implement the processes and methods of FIGS. 3a, 3b, and 4.
[0054] If programmable logic is used, such logic may execute on a commercially available processing platform configured by executable software code to become aspecific purpose computer or a special purpose device (e.g., programmable logic array, application-specific integrated circuit, etc.). A person having ordinary skill in the art may appreciate that embodiments of the disclosed subject matter can be practiced with various computer system configurations, including multi-core multiprocessor systems, minicomputers, mainframe computers, computers linked or clustered with distributed functions, as well as pervasive or miniature computers that may be embedded into virtually any device. For instance, at least one processor device and a memory may be used to implement the above described embodiments.
[0055] A processor unit or device as discussed herein may be a single processor, a plurality of processors, or combinations thereof. Processor devices may have one or more processor “cores.” The terms “computer program medium,” “non-transitory computer readable medium,” and “computer usable medium” as discussed herein are used to generally refer to tangible media such as a removable storage unit 618, a removable storage unit 622, and a hard disk installed in hard disk drive 612.
[0056] Various embodiments of the present disclosure are described in terms of this example computer system 600. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the present disclosure using other computer systems and / or computer architectures. Although operations may be described as a sequential process, some of the operations may in fact be performed in parallel, concurrently, and / or in a distributed environment, and with program code stored locally or remotely for access by single or multi-processor machines. In addition, in some embodiments the order of operations may be rearranged without departing from the spirit of the disclosed subject matter.
[0057] Processor device 604 may be a special purpose or a general-purpose processor device specifically configured to perform the functions discussed herein. The processor device 604 may be connected to a communications infrastructure 606, such as a bus, message queue, network, multi-core message-passing scheme, etc. The network may be any network suitable for performing the functions as disclosed herein and may include a local area network (LAN), a wide area network (WAN), a wireless network (e.g., WiFi), a mobile communication network, a satellite network, the Internet, fiber optic, coaxial cable, infrared, radio frequency (RF), or any combination thereof. Other suitable network types and configurations will be apparent to persons having skill in the relevant art. The computer system 600 may also include a main memory 608 (e.g., random access memory, read-only memory, etc.), and may also include a secondary memory 610. The secondary memory 610 may include the hard disk drive 612 and a removable storage drive 614, such as a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, etc.
[0058] The removable storage drive 614 may read from and / or write to the removable storage unit 318 in a well-known manner. The removable storage unit 618 may include a removable storage media that may be read by and written to by the removable storage drive 614. For example, if the removable storage drive 614 is a floppy disk drive or universal serial bus port, the removable storage unit 618 may be a floppy disk or portable flash drive, respectively. In one embodiment, the removable storage unit 618 may be non-transitory computer readable recording media.
[0059] In some embodiments, the secondary memory 610 may include alternative means for allowing computer programs or other instructions to be loaded into thecomputer system 600, for example, the removable storage unit 622 and an interface 620. Examples of such means may include a program cartridge and cartridge interface (e.g., as found in video game systems), a removable memory chip (e.g., EEPROM, PROM, etc.) and associated socket, and other removable storage units 622 and interfaces 620 as will be apparent to persons having skill in the relevant art.
[0060] Data stored in the computer system 600 (e.g., in the main memory 608 and / or the secondary memory 610) may be stored on any type of suitable computer readable media, such as optical storage (e.g., a compact disc, digital versatile disc, Blu-ray disc, etc.) or magnetic tape storage (e.g., a hard disk drive). The data may be configured in any type of suitable database configuration, such as a relational database, a structured query language (SQL) database, a distributed database, an object database, etc. Suitable configurations and storage types will be apparent to persons having skill in the relevant art.
[0061] The computer system 600 may also include a communications interface 624. The communications interface 624 may be configured to allow software and data to be transferred between the computer system 600 and external devices. Exemplary communications interfaces 624 may include a modem, a network interface (e.g., an Ethernet card), a communications port, a PCMCIA slot and card, etc. Software and data transferred via the communications interface 624 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals as will be apparent to persons having skill in the relevant art. The signals may travel via a communications path 626, which may be configured to carry the signals and may be implemented using wire, cable, fiber optics, a phone line, a cellular phone link, a radio frequency link, etc.
[0062] The computer system 600 may further include a display interface 602. The display interface 602 may be configured to allow data to be transferred between the computer system 600 and external display 630. Exemplary display interfaces 602 may include high-definition multimedia interface (HDMI), digital visual interface (DVI), video graphics array (VGA), etc. The display 630 may be any suitable type of display for displaying data transmitted via the display interface 602 of the computer system 600, including a cathode ray tube (CRT) display, liquid crystal display (LCD), light-emitting diode (LED) display, capacitive touch display, thin-film transistor (TFT) display, etc.
[0063] Computer program medium and computer usable medium may refer to memories, such as the main memory 608 and secondary memory 610, which may be memory semiconductors (e.g., DRAMs, etc.). These computer program products may be means for providing software to the computer system 600. Computer programs (e.g., computer control logic) may be stored in the main memory 608 and / or the secondary memory 610. Computer programs may also be received via the communications interface 624. Such computer programs, when executed, may enable computer system 600 to implement the present methods as discussed herein. In particular, the computer programs, when executed, may enable processor device 604 to implement the processes and methods illustrated by FIGS. 3a, 3b, and 4, as discussed herein. Accordingly, such computer programs may represent controllers of the computer system 600. Where the present disclosure is implemented using software, the software may be stored in a computer program product and loaded into the computer system 600 using the removable storage drive 614, interface 620, and hard disk drive 612, or communications interface 624.
[0064] The processor device 604 may comprise one or more modules or engines configured to perform the functions of the computer system 600. Each of the modules or engines may be implemented using hardware and, in some instances, may also utilize software, such as corresponding to program code and / or programs stored in the main memory 608 or secondary memory 610. In such instances, program code may be compiled by the processor device 604 (e.g., by a compiling module or engine) prior to execution by the hardware of the computer system 600. For example, the program code may be source code written in a programming language that is translated into a lower level language, such as assembly language or machine code, for execution by the processor device 604 and / or any additional hardware components of the computer system 600. The process of compiling may include the use of lexical analysis, preprocessing, parsing, semantic analysis, syntax-directed translation, code generation, code optimization, and any other techniques that may be suitable for translation of program code into a lower-level language suitable for controlling the computer system 600 to perform the functions disclosed herein. It will be apparent to persons having skill in the relevant art that such processes result in the computer system 600 being a specially configured computer system 600 uniquely programmed to perform the functions discussed above.
[0065] Techniques consistent with the present disclosure provide, among other features, systems and methods for authentication of a client device using a hash chain. While various exemplary embodiments of the disclosed system and method have been described above it should be understood that they have been presented for purposes of example only, not limitations. It is not exhaustive and does not limit the disclosure tothe precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practicing of the disclosure, without departing from the breadth or scope
Claims
WHAT IS CLAIMED IS:1 . A method for context extraction from media based on user voice input, the method comprising: receiving, by computer premises equipment (CPE ), a user voice input including a reference to an object in a piece of media content and an object request, the piece of media content being displayed on a display device connected to the CPE; capturing, by the CPE, a segment of the piece of media content upon receipt of the user voice input; analyzing, by the CPE, the captured segment for the object referenced in the user voice input; identifying, by the CPE, the object in the segment of the media content; and transmitting, by the CPE, an instruction to an external computing device, the instruction based on the object request.
2. The method of claim 1 , wherein the user voice input is received from a voice-enabled device.
3. The method of claim 1 , wherein the user voice input is received from a cloud server associated with a voice-enabled device.
4. The method of claim 1 , wherein the segment is a screenshot of the piece of media content at the time the user voice input is received.
5. The method of claim 1 , wherein the segment is a pre-determined length of video data associated with the piece of media content based on a time the user voice input is received.
6. The method of claim 1 , wherein the segment is a cache of all video data associated with the piece of media content prior to a time the user voice input is received.
7. The method of claim 1 , wherein the analyzing the captured segment for the object includes: extracting, by the CPE, data from the segment, wherein the data includes metadata or a QR code.
8. The method of claim 1 , wherein the identifying the object includes: identifying, by the CPE, a product name of the object.
9. The method of claim 1 , wherein the object request is a request to purchase a quantity of the object, the external computing device is an e-commerce server, and the instruction is a request to add the quantity of the object to an e-commerce shopping cart associated with an e-commerce account of the user.
10. The method of claim 1 , wherein the media content is one of: a live television program, a live movie, a live advertisement, a recorded television program, a recorded movie, and a recorded advertisement.11 . A system for context extraction from media based on user voice input, the system comprising: one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage devices, and instructions stored on at least one of the one or more storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, the instructions comprising: instructions to receive a user voice input including a reference to an object in a piece of media content and an object request, the piece of media content being displayed on a display device; instructions to capture a segment of the piece of media content upon receipt of the user voice input; instructions to analyze the captured segment for the object referenced in the user voice input; instructions to identify the object in the segment of the media content; and instructions to transmit an instruction to an external computing device, the instruction based on the object request.
12. The method of claim 11 , wherein the user voice input is received from a voice- enabled device.
13. The method of claim 11 , wherein the user voice input is received from a cloud server associated with a voice-enabled device.
14. The method of claim 11 , wherein the segment is a screenshot of the piece of media content at the time the user voice input is received.
15. The method of claim 11 , wherein the segment is a pre-determined length of video data associated with the piece of media content based on a time the user voice input is received.
16. The method of claim 11 , wherein the segment is a cache of all video data associated with the piece of media content prior to a time the user voice input is received.
17. The method of claim 11 , wherein the analyzing the captured segment for the object includes:Extracting data from the segment, wherein the data includes metadata or a QR code.
18. The method of claim 11 , wherein the identifying the object includes: identifying a product name of the object.
19. The method of claim 11 , wherein the object request is a request to purchase a quantity of the object, the external computing device is an e-commerce server, and theinstruction is a request to add the quantity of the object to an e-commerce shopping cart associated with an e-commerce account of the user.
20. A computer program product for context extraction from media based on user voice input, the computer program product comprising: a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method, comprising: receiving, by consumer premises equipment (CPE), a user voice input including a reference to an object in a piece of media content and an object request, the piece of media content being displayed on a display device connected to the CPE; capturing, by the CPE, a segment of the piece of media content upon receipt of the user voice input; analyzing, by the CPE, the captured segment for the object referenced in the user voice input; identifying, by the CPE, the object in the segment of the media content; and transmitting, by the CPE, an instruction to an external computing device, the instruction based on the object request.
Citation Information
Patent Citations
Media Presentation Device with Voice Command Feature
US20200196018A1
Adding audio and video context to smart speaker and voice assistant interaction
US20200380981A1
