Automatic content identification fingerprint sequence matching

Abstract

The present application relates to automatic content recognition fingerprint sequence matching. A media device and method execute computer instructions to: generate a first query fingerprint for media content; send the first query fingerprint to a server; and receive from a server an identification of a first channel being viewed in response to the server finding a match with the first query fingerprint. The media device repeatedly performs the following operations: receives from the server a plurality of fingerprints generated from locations of the media content that are located before where the media content is being received; stores the plurality of fingerprints in a local database on a computer-readable storage medium; and compares a second query fingerprint for the media content to the plurality of fingerprints to determine whether the media device remains set to the first channel without continuously retrying the server. When the media device detects a mismatch with the second query fingerprint locally, the process restarts with it looking for a match at the server.

Description

[0001] Information about divisional applications

[0002] This application is a divisional application of the Chinese invention patent application with application number 201680023388.4, application date April 21, 2016, and invention name “Automatic Content Identification Fingerprint Sequence Matching”. Technical Field

[0003] The present application relates to automatic content recognition fingerprint sequence matching. Background Art

[0004] Media consumption devices, such as smart televisions (TVs), can access broadcast digital content and receive data (e.g., streaming media) from data networks (e.g., the Internet). Streaming media refers to services in which media content (e.g., movies or news) is provided to end users upon request via telephone lines, cables, the Internet, etc. For example, users can watch movies without leaving their homes. Furthermore, users can access various types of educational content (e.g., video lectures) without having to physically visit a school or educational institution.

[0005] As the number of media consumption devices continues to increase, video content production and delivery may similarly increase. With the increasing use of media consumption devices (such as smartphones, tablets, and smart TVs) to access streaming media, content or network providers (such as local broadcast stations, multi-channel networks, and other content owners / distributors) may distribute contextually relevant materials to viewers who are consuming streaming media (e.g., media programs). For example, a local broadcast station may include contextually relevant advertisements and content that interacts with the streaming media. Summary of the Invention

[0006] In one aspect, the present application provides a media device comprising: at least one processing device; and a non-transitory computer-readable medium storing program instructions that, when executed by the at least one processing device, cause the media device to perform operations comprising: generating a first query fingerprint for first media content being consumed by the media device; sending the first query fingerprint to a server; and in response to the server finding a match with the first query fingerprint: (i) receiving from the server an identification of a first channel being viewed via the media device; (ii) receiving from the server an identification of an upcoming media content from a first channel being viewed; The media device is configured to: (i) generate a first plurality of fingerprints for a second upcoming media content to be displayed on the first channel after the first media content; and (ii) receive a second plurality of fingerprints from the server for a second upcoming media content to be displayed on a second channel different from the first channel; store the first plurality of fingerprints and the second plurality of fingerprints in a local database of the media device; generate a second query fingerprint for a second media content being consumed by the media device; compare the second query fingerprint with the first plurality of fingerprints and the second plurality of fingerprints stored in the local database; and determine, based on the comparison of the second query fingerprint, whether the media device remains set to the first channel or has changed to the second channel.

[0007] On the other hand, the present application provides a method, which includes generating, by a media device, a first query fingerprint for first media content being consumed by the media device; sending, by the media device, the first query fingerprint to a server; in response to the server finding a match with the first query fingerprint: (i) receiving, from the server, an identification of a first channel being viewed via the media device; and (ii) receiving, from the server, a first plurality of fingerprints generated from first upcoming media content, which will be displayed on the first channel after the first media content; and (iii) receiving, from the server, a second plurality of fingerprints of second upcoming media content to be displayed on a second channel different from the first channel; storing, by the media device, the first plurality of fingerprints and the second plurality of fingerprints in a local database of the media device; and generating a second query fingerprint for second media content being consumed by the media device; comparing the second query fingerprint with the first plurality of fingerprints and the second plurality of fingerprints stored in the local database; and determining, based on the comparison of the second query fingerprint, whether the media device remains set to the first channel or has changed to the second channel. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention will be more fully understood from the detailed description given below and from the accompanying drawings of various embodiments of the present invention. However, the drawings should not be considered to limit the present invention to specific embodiments, but are for explanation and understanding purposes only.

[0009] Figure 1 Illustrated is a system diagram of a content distribution network with client-side automatic content recognition (ACR) fingerprint local matching according to one embodiment.

[0010] Figure 2 Illustrates a content manager in communication with an ACR client that performs fingerprint matching on a client media device, according to one embodiment.

[0011] Figure 3 Illustrated is a system diagram of a content distribution network with client-side ACR fingerprint local matching according to one embodiment.

[0012] Figure 4 A flow chart illustrating a method of media device fingerprinting of media content and local matching of fingerprints according to one embodiment.

[0013] Figure 5 Illustrated is a flow chart illustrating a method of media device fingerprinting of media content and local matching of fingerprints according to another embodiment.

[0014] Figure 6 The illustration illustrates a diagrammatic representation of a machine in the example form of a computer system within which a set of instructions, for causing the machine to perform any one or more of the methodologies discussed herein, may be executed. DETAILED DESCRIPTION

[0015] A media content broadcast or stream (e.g., a television (TV) or Internet show broadcast) can be an attractive venue for advertising products and services, providing information to viewers, or any combination thereof. Thus, broadcasters would like to know what programs (or channels) individual viewers are watching and when, so that the themes of those programs (and the timing of commercials) can be used to accurately target advertisements and other useful, optionally non-commercial information to those viewers. For example, non-commercials can include news alerts, announcements, or educational information. Therefore, it would be advantageous to determine the programs a user is watching or is about to watch, and to send an identification of the program information to the user's media device or to an advertising server for such targeted action.

[0016] To make this determination, automatic content recognition (ACR) can be used to generate query fingerprints from the media content (e.g., a television program or channel) that the user is watching. These query fingerprints can be compared with fingerprints stored in a database to find a match. This database is historically stored at a server (e.g., an ACR server) to which the fingerprints are sent to find one or more matches, so that an identification of a channel can be generated, from which the television program currently being streamed (or will soon be streamed) can be determined. The server then sends the identification of the media content (e.g., a television program) to a requesting entity (e.g., a smart TV or an advertising server), which can use the information to target the user with advertisements during commercial breaks or as an overlay on the media content (which may or may not occur during commercial breaks).

[0017] Because channel awareness is time-sensitive, as commercial slots vary from channel to channel, media devices (such as televisions or set-top boxes) continually query servers (for example, every second or every few seconds). As the number of media devices streaming media content grows, there is an exponential increase in the number of queries that the server must make to confirm which channel each user is watching. This places a significant burden on the server in terms of bandwidth and processing power. The present invention seeks to alleviate this need for constantly querying servers.

[0018] For example, a media device and method may include: a computer-readable storage medium that stores instructions; and a processing device that executes the instructions to generate a first query fingerprint for media content that a user is consuming. The media device sends the first query fingerprint to a server and receives from the server an identification of a first channel that the user is watching in response to the server finding a match with the first query fingerprint. The media device may continuously receive from the server multiple fingerprints generated from positions of the media content that are located before where the user is watching the media content, and store the multiple fingerprints in a local database of the computer-readable storage medium. The media device may then compare a second (and any subsequent) query fingerprint of the media content with the multiple fingerprints to determine that the user has not changed to a second channel that is different from the first channel. As long as the local database continues to return matches with the same channel, any further queries to the server are unnecessary.

[0019] However, when the local database returns an answer that any subsequent (or second) fingerprint does not match the same channel, the media device can query the server again to determine the new channel to which the user has changed the media device. Once the new channel is known, the server can continue to send fingerprints (again, generated before the point the user was watching) to the media device for local storage. The media device can then continue to confirm that the user remains on the new channel by comparing subsequent fingerprints with the newly stored fingerprints in the local database.

[0020] Furthermore, in addition to multiple fingerprints of television programs or channels, the media device may also continuously receive from the server multiple fingerprints of media content from nearby channels (e.g., sequentially nearby the channel the user is currently watching) and / or multiple fingerprints from channels the user frequently watches. When a local match occurs, the media device can still locally look for a match within the fingerprints of nearby channels or one of the user's favorite channels (even after the user changes channels), thereby avoiding the need to query the server again. In this way, knowledge of the user's viewing behavior can be combined with local matching to minimize bandwidth and processing burden on the server.

[0021] The media device (or server) may send an identifier of the channel that identifies the television program being viewed by the user to a lookup server. The lookup server may then notify an ad server of the television program and receive from the ad server an advertisement that is contextually relevant to the subject matter of the media program. The lookup or ad server may then deliver the advertisement (or other content) to the media device for display as an overlay or replacement advertisement (which may also be an informational snippet) during a commercial break in the media program. Alternatively, the overlay may be overlaid on top of a portion of the television program (which may be referred to as the subject of the overlay advertisement).

[0022] To provide media content to viewers, an individual or organization may decide to stream the media content to the viewer, for example, by delivering the media content to the viewer via the Internet. The media content used by the individual or organization may be media content (e.g., video data) obtained from one or more real-time broadcast media feeds. For example, a media content provider may provide a linear media channel (e.g., media provided to the viewer from a real-time media feed) to a user via the Internet.

[0023] The term "content" may be used to refer to media or multimedia. The term "content" may also be a specific term that refers to the subject matter of the media rather than the media itself. Similarly, the term "media" and some compound terms containing "media" (e.g., multimedia, hypermedia) refer instead to the content rather than the channel through which the information is delivered to the end user / viewer. Media or media content may include graphical representations, such as videos, movies, television shows, commercials, streaming video, etc., text, graphics, animations, still images, interactive content formats, etc. An example of a type of content that is commonly referred to as a media type is a "moving picture" called a "movie."

[0024] In one embodiment, a content overlay system or device may enable the combination of media content with specific real-time and / or targeted overlay content (e.g., advertisements). The content overlay system or device may enable overlay content providers to engage viewers by inviting them to respond to a call to action within the content overlay (e.g., an invitation to participate in the content overlay). One advantage of inviting viewers to participate in the call to action may be providing a return path or follow-up path for the viewer to request additional information, ask questions, provide input, contact the provider of the advertised service or product, and so on. Another advantage of inviting viewers to participate in the call to action may be providing a return path or follow-up path for the advertiser to provide additional information, further engage the viewer, gather additional information about the viewer, answer the viewer's questions about the advertised product or service, and so on. In another example, once a viewer has viewed and / or interacted with the overlay content of a media program, the content overlay system or device may enable advertisers to use cross-platform retargeting campaigns.

[0025] Figure 1 A system diagram of a content distribution network 100 according to one example is illustrated. In the content distribution network 100, a content provider 102 may broadcast a content feed to a local provider 106. The local provider 106 may include a headend 104 and an automatic content recognition (ACR) fingerprint reader server 105. The content feed from the content provider 102 may be received at the headend 104 of the local provider 106. The headend 104 may generate a local content feed based on the received content feed. For example, the headend 104 may be a local affiliate radio station that receives a network channel with programming and advertisements from a national or global radio station.

[0026] The headend 104 may communicate the local content feed to an ACR fingerprinter server 105, an over-the-air (OTA) broadcaster 108, and / or a multichannel video programming distributor (MVPD) 110. The OTA broadcaster 108 and / or the MVPD 110 may communicate the local content feed to a media device 115. Some examples of media devices include client devices 118 and 120, a set-top box 114 that streams provider content to the client devices 118 and 120, and other devices 116 through which a user may stream (e.g., wirelessly) the local content feed.

[0027] In one example, OTA broadcast station 108 may broadcast a local content feed using traditional local television or radio channels. In this example, client devices 118 and 120 may include an antenna (e.g., a TV or radio antenna) and receive the local content feed. In another example, MVPD 110 (e.g., a cable or satellite broadcast station) may communicate the local content feed to set-top box 114. In this example, set-top box 114 may format the content feed for client devices 118 and 120 and may communicate the formatted content feed to client devices 118 and 120. Client devices 118 and 120 may include a display device, such as a television screen or touch screen, to display local content to viewers. Various components of content distribution network 100 may be integrated or coupled to client devices 118 and 120. For example, a smart TV may include an antenna, set-top box 114, and a display device in a single unit.

[0028] The ACR fingerprint server 105 may analyze the local content feed and determine fingerprint information (eg, fingerprint). The ACR fingerprint server 105 may communicate the fingerprint to the ACR system 124 or 126. The ACR systems 124 and 126 may be different ACR systems selected by the device manufacturer (eg, smart TV manufacturer).

[0029] In some embodiments, automatic content recognition may be performed in a distributed architecture as described herein. In these embodiments, an ACR client 125 (e.g., a local ACR system or a local ACR fingerprint matcher) may reside on any media device 115. The ACR client 125 may perform ACR fingerprint local matching as described herein. Figure 2 and 3 Describe in more detail the ACR client 125 and the ACR fingerprint server 105 (or Figure 1 or Figure 2 interaction with other systems).

[0030] The ACR fingerprint server 105 can analyze the local content feed and capture fingerprints, which may include an ordered sequence of frames from the local content feed. The ACR fingerprint server 105 can communicate the fingerprints to the ACR system 124 or 126. The ACR systems 124 and 126 can be different ACR systems selected by the device manufacturer (e.g., a smart TV manufacturer). In one example, the ACR fingerprint server 105 can format the fingerprints for different ACR systems 124 and 126. The ACR systems 124 and 126 can establish communication connections with different media devices 115. The media device 115 can communicate the fingerprint information to the ACR systems 124 and 126, respectively. When the ACR system 124 or 126 receives the ACR fingerprint information from the media device 115, the ACR system 124 or 126 can match the received fingerprints with those generated by the ACR fingerprint server 105. When a match occurs and the content has been identified, the ACR system 124 or 126 can communicate the ACR event to the content manager 122.

[0031] In another example, the ACR system 124 or 126 may receive ACR fingerprint information from the media device 115 and may match the received fingerprints with those generated by the ACR fingerprint server 105. When a match occurs and content has been identified, the ACR system 124 or 126 may notify the media device 115 of an ACR event, and the media device 115 may then communicate those ACR events to the content manager 122. Alternatively or in addition, the ACR system 124 or 126 may communicate the ACR events directly to the content manager 122. The ACR fingerprint information may include: the display of an advertisement in the viewer's local content feed, the display of selected or tagged content in the viewer's local content feed, a change of content channel at the media device 115, and so on.

[0032] The event information from different ACR systems 124 and 126 may be in different formats, and the content manager 122 may normalize the data into a common format before storing the data in the database 123. For example, the content manager 122 may receive different data sets containing similar, but not identical, data (e.g., data having the same content but formatted in different ways) from the ACR systems 124 and 126. The content manager 122 may process and reformat the different data sets to create a single data model or format (e.g., a reformatted data set), and the reformatted data set may be populated into the database 123 in the content manager 122.

[0033] In one embodiment, in order to normalize the different data sets from the ACR systems 124 and 126, the content manager 122 may clean or filter the data in the data sets. For example, some data sets may contain fields or data that may be irrelevant to the content manager 122. In this example, the content manager 122 may clean or filter the irrelevant data (e.g., data that can be removed or ignored). In another example, some data sets may include examples of incomplete or incorrect data or data sets, and the content manager 122 may clean or filter the incomplete or incorrect data or data sets. In another embodiment, in order to normalize the different data sets from the ACR systems 124 and 126, the content manager 122 may map the fields of the data sets. For example, when the content manager 122 receives a first data set from the ACR system 124 and a second data set from the ACR system 126, at least some of the data fields of the first data set and the second data set may be common to both the first and second data sets. However, the common data fields may be located at different locations in the first and second data sets. In this example, the content manager 122 may map the different data fields of the first and second data sets to normalized fields and have the same data fields in the same data field locations in the database 123 .

[0034] In another embodiment, the content manager 122 may derive data from the datasets to normalize the different datasets from the ACR systems 124 and 126. For example, the data from the ACR systems 124 and 126 may not contain all the fields needed to populate the data fields in the database. However, the content manager 122 may use other fields in the datasets from the ACR systems 124 and 126 to derive data for these data fields.

[0035] In one example, database 123 may include data fields such as state, designated market area (DMA), and county and / or city fields within a country field, but the data sets from ACR systems 124 and 126 may only include zone improvement program (ZIP) codes. In this example, content manager 122 may use the ZIP codes to derive data for the fields in the database. In another example, the data sets may not contain any geographic location information but may include the Internet Protocol (IP) addresses of ACR systems 124 and 126. In this example, content manager 122 may use a geo-IP lookup service to derive state, DMA, county, city, and ZIP code information.

[0036] In another example, database 123 may include demographic fields, such as an age field, a gender field, a household income field, and the like. However, the data sets from ACR systems 124 and 126 may not include demographic fields or demographic data. In this example, ACR systems 124 and 126 may provide content manager 122 with the IP address of media device 115. Content manager 122 may use the IP address to determine demographic data to populate the data fields in the database.

[0037] In another example, a field in a first data set from the ACR system 124 may include local time zone information (e.g., Mountain Daylight Time (MDT) zone), and a second data set from the ACR system 126 may include information from another time zone (e.g., Coordinated Universal Time (UTC) zone). The database may store all data using UTC, and the content manager 122 may convert the local time to UTC before storing the data in the database 123.

[0038] In one embodiment, the content manager 122 may use the normalized data to generate reports or data (viewing data) about user viewing behavior across different ACR technology vendors and smart TVs or other Internet-connected video devices. The content manager 122 and the media device 115 may include a communication interface for communicating information (e.g., overlay content) between the media device 115 and the content manager 122. In one example, the communication interface may communicate information using a cellular network and / or a wireless network. In one example, the communication network may be a cellular network that utilizes Third Generation Partnership Project (3GPP) Release 8, 9, 10, 11, or 12 or the Institute of Electrical and Electronics Engineers (IEEE) 512 standard. 802.16p, 802.16n, 802.16m-2011, 802.16h-2010, 802.16j-2009, 802.16-2009. In another embodiment, the communication network may be compliant with the IEEE 802.16 standard developed by the Institute of Electrical and Electronics Engineers, Inc. Wireless networks that use standards (such as IEEE 802.11-2012, IEEE802.11ac, or IEEE 802.11ad) developed by the Wi-Fi Alliance In another embodiment, the communication network may deploy the Bluetooth technology developed by the Bluetooth Special Interest Group (SIG). In another embodiment, the communication network may be a Bluetooth v1.0, Bluetooth v2.0, Bluetooth v3.0 or Bluetooth v4.0 network developed by the ZigBee Alliance. Connection, such as IEEE 802.15.4-2003 (Zigbee 2003), IEEE802.15.4-2006 (Zigbee 2006), IEEE 802.15.4-2007 (Zigbee Pro).

[0039] In one example, the content manager 122 may also instruct the media device 115 to replace a portion of a local content feed received from the OTA broadcaster 108 or the MVPD 110 with the overlay content. In another example, the content manager 122 may instruct the media device 115 to overlay or superimpose the overlay content onto a portion of the local content feed. The content manager 122 may aggregate ACR information across multiple ACR systems 124 and 126 and may communicate the overlay content to different media devices 115 (e.g., client devices from different device manufacturers).

[0040] The content manager 122 may also establish a communication connection with the other devices 116. In one example, the other devices 116 may communicate with the client devices 118 and 120 and provide an additional screen (e.g., a second screen) to display overlay content. For example, the client devices 118 and 120 may receive a local content feed from the OTA broadcast station 108 or the MVPD 110 and display the local content feed to the user. When an ACR event occurs, the other devices 116 may also communicate the ACR event information to the ACR systems 124 and 126, as discussed in the previous paragraph. When the content manager 122 receives the ACR event information, the content manager 122 may communicate the overlay content to the other devices 116.

[0041] In one example, client devices 118 and 120 may continue to display the local content feed while other devices 116 display the overlay content. In another example, both client devices 118 and 120 and other devices 116 may display the overlay content. In another example, client devices 118 and 120 and other devices 116 may display a portion of the overlay content and a portion of the local content feed. In another example, client devices 118 and 120 and other devices 116 may display different local content feeds and / or overlay content.

[0042] In one example, client devices 118 and 120 and / or other devices 116 may display the overlay content upon receiving it. In another example, client devices 118 and 120 and / or other devices 116 may delay displaying the overlay content for a threshold period of time. The threshold period of time may be a predefined period of time, or content manager 122 may select a period of time for which client devices 118 and 120 and other devices 116 delay displaying the overlay content.

[0043] Figure 2The diagram illustrates a content manager 122 in communication with an ACR client 125 of a media device 115 that performs fingerprint matching locally on the media device, according to one embodiment. A content provider 202 may stream media content to the media device 115 via a network 219, which may be intercepted by the ACR content manager 122 prior to or concurrently with the streaming of the media content to the media device 115. The content manager 122 may also communicate with an advertising server (or "ad" server) 230, for example, sending an ad call to the ad server 230 requesting that an advertisement be provided with (or as an overlay on) the media content to target the subject matter of the media content and / or the user's interests, as will be explained in greater detail.

[0044] The content manager 122 may include an ACR engine 204, a lookup server 206, an overlay decision engine 210, and an advertisement targeter 212. A content provider 202 may upload media content to the ACR engine 204. The ACR engine 204 may fingerprint the media content. In one example, a fingerprint may be generated by fingerprinting each frame of a feed, every other frame of a feed, a sequence of frames, and so on. For example, the ACR engine 204 may generate a fingerprint for a frame of a feed by performing a discrete cosine transform (DCT) on the frame and designating a subset of the resulting coefficients (e.g., low-frequency coefficients) as the fingerprint.

[0045] Further references Figure 2 , the ACR engine 204 can communicate the ACR fingerprints to the lookup server 206, which can look up and determine the television program (or the like) and the location within the television program that corresponds to the ACR event received from the ACR system 124 or 126. Each fingerprint of a segment of the feed can be associated with a timestamp. The timestamp can be the individual frame of the segment of the feed at the time it is received by the ACR engine 204. The timestamp can be a frame number from an arbitrary starting point within the feed. The lookup server 206 can store the fingerprints in association with their corresponding timestamps and help the advertising targeter 212 and the overlay decision engine 210 time and target the content that the user is viewing within the media content of the feed.

[0046] In one embodiment, the ACR engine 204 interacts with various media devices 115 ( Figure 1 ) at the ACR client 125. The ACR client 125 can match the fingerprint locally and confirm whether the user has changed the channel to watch a different TV program. Figure 3 Details about the ACR client 125 are described.

[0047] The media device 115 may include an ACR client (also referred to herein as an ACR client 125) that may periodically, continuously, or semi-continuously communicate user fingerprint information to the lookup server 206. The lookup server 206 may determine when a match may exist between ACR fingerprint information (e.g., stored at the lookup server 206) and the user fingerprint information. When a positive match exists, the lookup server 206 may communicate a positive match indicator to the ACR client 125. The ACR client 125 may send an overlay request requesting an overlay of the media content to the overlay decision engine 210.

[0048] In one example, the overlay request may include a media content identifier (ID). In another example, the overlay request may include overlay information or overlay parameters (as discussed in the previous paragraph). In one example, the overlay decision engine 210 may use content ID, overlay information, and / or overlay parameters to identify the target overlay content. In another example, the overlay decision engine 210 may use content ID, overlay information, and / or overlay parameters to identify the overlay format. The overlay decision engine 210 may compare the content ID, overlay information, and / or overlay parameters with the overlay database to identify the target overlay content and / or overlay format. The overlay database may be updated with new overlay content and / or overlay formats by content providers or advertisers on a periodic or continuous basis.

[0049] When the overlay decision engine 210 identifies the target overlay content, the overlay decision engine 210 may transmit the target overlay content back to the media device 115. In one example, the overlay decision engine 210 may transmit the target overlay content directly to the media device 115 (e.g., via a wireless communication network). In another example, the overlay decision engine 210 may transmit the target overlay content to the media device 115 via a uniform resource locator (URL). In one example, when multiple target overlay contents match the content ID, overlay information, and / or overlay parameters, the overlay decision engine 210 may select the target content overlay that meets the greatest number of parameters and / or other information (e.g., is greater than the other overlay contents). In another example, when multiple target overlay contents match the content ID, overlay information, and / or overlay parameters, the overlay decision engine 210 may randomly select the overlay content that meets the parameters and other information. In another example, when multiple target overlay contents match the content ID, overlay information, and overlay parameters, the overlay decision engine 210 may select a predetermined overlay content that matches the content ID, overlay information, and / or overlay parameters. The overlay content can be populated with dynamic content (eg, content that can be updated or refreshed at periodic intervals). The dynamic content can be stored in a local database or an external system.

[0050] The ACR client 125 of the media device 115 can overlay the overlay content on the content feed when the ACR fingerprint information matches the user fingerprint information. In one example, the client device 208 can overlay the overlay content on the content feed in a hypertext markup language (HTML) browser. In another example, the client device 208 can overlay the overlay content on the content feed from an OTA broadcast station or a cable broadcast station. When the overlay content is placed on the content feed, the overlay content can be displayed to the user via the display of the media device 115. In one example, the overlay content may include one or more calls to action options that can be displayed to the user. In this example, the user can use an input device (such as a TV remote control, keyboard, smart phone or tablet computer) to interact with the overlay content to create feedback information. The ACR client 125 can communicate the feedback information to the advertising target determiner 212. Another entity (such as an advertiser) can access the feedback information and analyze the feedback information to determine the desired information, such as the user's focus in the overlay content.

[0051] The ACR client 125 may monitor the content feed to determine when the overlay content ceases to match the content feed and / or when a threshold time period expires. In one example, when the overlay content ceases to match the content feed and / or the threshold time period expires, the media device 115 may cease superimposing the overlay content for display on the media device 115.

[0052] In other embodiments, the ACR client 125 may perform some fingerprint matching locally for some or all of the fingerprint matching. That is, the ACR client may perform these calculations locally on the client device 208 rather than on a server (e.g., the lookup server 206). It should be noted that the ACR operations described herein need not be used in the context of an overlay system. That is, the ACR fingerprint local matching performed by the ACR client 125 may be used in other systems, fewer or more than those described in the context of an overlay.

[0053] Figure 3 The content preparation system 300 with client-side ACR fingerprint local matching according to one embodiment is illustrated. The content preparation system 300 may include a content device 302 (such as an appliance or other device), an ACR cloud 310 (such as the one described above with respect to Figure 1 and Figure 2315 . The content device 302 may include a capture daemon 304, a capture card 306, and a software updater 308. The capture daemon 304 may further include a capturer 314, a segment encoder 316, and an uploader 318 within a processing chain, as well as content frames 320 and an ACR fingerprinter 322 for fingerprinting the media content captured by the capturer 314. The content device 302 may receive media content (also referred to herein as simply "content") from a content feed and prepare the content to include contextually relevant material that can be delivered to a media device 315 that consumes the media content for viewing by a user. The media device 315 may include an ACR client 325 for performing local matching of ACR fingerprints. The content feed may be received using the capture card 306.

[0054] The capture card 306 can be used to capture and convert media content (e.g., audio and / or video content) from a content feed, such as a stream encoded using HD-SDI (High-Definition Serial Digital Interface) developed by the Society of Motion Picture and Television Engineers. In one example, the capture card 306 can receive the content feed using a coaxial cable port. In another example, the capture card 306 can receive the content feed using a High-Definition Multimedia Interface (HDMI) port. The capture card 306 may include a serial digital interface (SDI) port. An SDI port can be configured to receive content (input port) or send content (output port). The capture card 306 can use the SDI port to receive one or more content feeds. When the capture card 306 receives multiple content streams, each content feed can be fingerprinted, encoded, and uploaded as previously discussed. For example, the capture card 306 can receive two content feeds using SDI IN 1 and SDI IN 2, respectively. In this example, the capture daemon 304 can process the content feeds in parallel.

[0055] When the capture card 306 receives content on a content feed, the capture box can send the content to the capture box 314. The capture box 314 can be software used to control and interact with the capture card 306. The capture box 314 can instruct the capture card 306 to use the SDI port as input and / or instruct the capture card 306 on how to obtain incoming media content, such as audio / video ("A / V") data. The capture box 314 can use the capture card 306 to obtain advanced media content information and perform advanced functions (e.g., closed captioning information, time code delivered in the feed) and adjust the media content data, such as color space conversion or scaling.

[0056] When the capturer 314 receives content from the capture card 306, the capturer 314 can provide the content to both the ACR fingerprint reader 322 and the encoder 316. For example, the capturer 314 can provide A / V data to both the ACR fingerprint reader 322 and the encoder 316, e.g., splitting the content for delivery to the ACR fingerprint reader 322 and the encoder 316. In one example, the capturer 314 can communicate with the broadcast cloud 312 to indicate when media content is received via the capture card 306 for upload to the broadcast cloud 312, e.g., a streaming status.

[0057] In another example, the capturer 314 can receive a status message from the broadcast cloud 312 indicating the status (e.g., reception status) of the broadcast cloud 312 receiving streaming content from the capture daemon 304. For example, the capture daemon 304 can use a polling method to communicate with the broadcast cloud 312. In this example, the capture daemon 304 can send a status message to inform the broadcast cloud 312 of the streaming status of the capture daemon 304 (e.g., whether the content is ready for streaming or there is no current content for streaming), and can additionally retrieve the reception status from the broadcast cloud 312.

[0058] In yet another example, the capturer 314 may provide system metric information to the broadcast cloud, such as central processing unit (CPU) utilization, memory utilization, drive utilization, operating temperature, etc. In another example, the software updater 308 may be configured to update the content device 302. For example, the content device 302 may use a polling method to communicate with the broadcast cloud 312 to determine when an update is scheduled.

[0059] A segment encoder 316 (hereinafter encoder 316) may receive content from the capturer 314 and encode the content to provide secure content for transmission between the broadcast cloud 312 and a media device 315 (e.g., client device 118 or 120). The encoder 316 may also encode or convert the data to adapt to different display sizes, processing capabilities, network conditions, and / or battery capacities of the media consumption device. The encoder 316 may write the content segments to the encoder's 316 disk or memory. Once the content has been encoded, the encoder 316 may then transmit the encoded content to the uploader 318. After the content has been transmitted to the uploader 318, the content segments may be deleted from the encoder's 316 disk or memory. Once the uploader 318 receives the encoded content, the uploader 318 may upload the encoded content or segments of the encoded content to the broadcast cloud 312. When the uploader 318 has uploaded the encoded content, the uploader 318 may notify the broadcast cloud 312 that the encoded content has been encoded and provide content metadata, such as the time the content was captured by the capture card 306 and / or the duration of the media content.

[0060] The broadcast cloud 312 may process the uploaded encoded content or segments before delivering the content or segments for playback at the media device 315. Such processing by the broadcast cloud may include transcoding the content or segments into multiple bit rates to support adaptive bit rate playback, and uploading all resulting segments to a content delivery network (CDN).

[0061] In parallel with delivering the content to the encoder 316, the capturer 314 may deliver content frames 320 to the ACR fingerprint reader 322. In one example, the content frames 320 may be audio data. In another example, the content frames 320 may be video data. In this example, the video content may consist of raw video frames. When the ACR fingerprint reader 322 receives the content frames 320, the ACR fingerprint reader 322 may determine how to process the content frames 320, such as how to process the raw video frames.

[0062] In one example, video frames may be fingerprinted individually. In another example, video frames may be fingerprinted in a collective or sequential manner. The ACR fingerprint reader 322 may determine when to fingerprint video frames individually or sequentially based on the ACR algorithm used by the ACR fingerprint reader 322. In another example, the ACR fingerprint reader 322 may fingerprint content frames 320 in different ways for different broadcasters or users. In this example, the ACR fingerprint reader 322 may include different ACR fingerprinting algorithms for different ACR vendors. In one example, the different ACR fingerprinting algorithms may be predetermined and stored on the memory of the ACR fingerprint reader 322. In another example, the different ACR fingerprinting algorithms may be provided by a third-party ACR vendor. When different ACR fingerprinting algorithms are provided by a third-party ACR vendor, the ACR fingerprint reader 322 may aggregate the different ACR fingerprinting algorithms. In one example, ACR fingerprinting may use original video in a YUV4:2:2 color space and at a high resolution. When video content is received at the capture card 306 , the capturer 314 may convert the video content to a YUV 4:2:0 color space and downscale its resolution to a threshold resolution level for encoding by the encoder 316 .

[0063] When the ACR fingerprint reader 322 has fingerprinted the content frame 320 of the media content, the ACR fingerprint reader may send the fingerprint (including the channel information, time code and fingerprint information) to the ACR cloud 310, which may be any local or remote server, such as the reference Figure 1 and 2As discussed above. In one example, different ACR fingerprinting algorithms can be used for the same media content to provide different fingerprint information to the ACR Cloud 310 of different ACR vendors. The advantage of fingerprinting the same content can be providing contextually relevant advertising and interactive content to different viewers of media consumption devices (such as media device 315). In another example, the content frame 320 can include media content from different feeds. In this example, different ACR fingerprinting algorithms can be used for content from different feeds to provide different fingerprinting information to the ACR Cloud 310 of different ACR vendors.

[0064] Different fingerprint identification information can be uploaded to the ACR cloud 310 of different ACR suppliers. Different ACR suppliers can be integrated on viewing devices manufactured by different contract equipment manufacturers (CEMs). TVs manufactured by Televisions manufactured using Fingerprinting of Resulting ACR Technology. An advantage of the fingerprinter 322 employing ACR fingerprinting algorithms of different ACR vendors may be that content can be fingerprinted and provided to viewers via different ACR vendors regardless of the manufacturer of the media consumption device.

[0065] In one example, the ACR cloud 310 can use the ACR fingerprint information for digital ad replacement (DAR). In another example, the ACR fingerprint information can be used for advertising or content enhancement and data collection. The fingerprint information can be sent to the ACR cloud 310, where the ACR cloud 310 can use the fingerprint information to match the encoded content with contextually relevant advertisements and interactive content. The matched encoded content, contextually relevant advertisements, and interactive content can then be provided to the media device 315 for display. In another example, information about the matching event can be communicated to the broadcast cloud 312 to analyze content matching.

[0066] In one example, ACR fingerprinting information can be delivered to the ACR Cloud 310 before the encoded segment can be delivered from the broadcast cloud 312 to the ACR Cloud 310. In another example, the encoded segment can be uploaded to the ACR Cloud 310 after the entire segment has been encoded. In another example, content can be fingerprinted continuously, and the resulting fingerprinting information can be delivered to the ACR provider cloud continuously or at a selected rate. The selected rate can be based on different ACR providers. For example, one provider may deliver ACR fingerprinting information several times per second, while another provider may deliver this information once every 2-5 seconds.

[0067] In one example, the content device 302 can encode, encrypt, and upload the media content to the broadcast cloud 312. In another example, the content device 302 can receive broadcast station-specific schedule metadata (data about what shows and advertisements are in the content feed received at the capture card 306) from the broadcast station. The content device 302 can communicate the metadata to the broadcast cloud 312. In another example, the broadcast cloud 312 can communicate the metadata (e.g., broadcast station-specific schedule metadata) to the ACR cloud 310 or a different ACR cloud 310 of a different vendor so that the vendor can use the metadata to match against the fingerprint information received from the ACR fingerprint reader 322.

[0068] An advantage of the content preparation system 300 that provides encoding, uploading, and fingerprinting is that it provides fault-tolerant content streaming along with ACR fingerprinting. For example, when the ACR fingerprint reader 322 fails, content streaming can continue without interruption and ACR fingerprinting can be restarted. In one example, the content preparation system 300 can perform ACR fingerprinting as a separate process from content encoding and uploading to avoid failures when providing streaming content for viewing.

[0069] In one example, the ACR fingerprinter 322 may perform ACR fingerprinting upstream of broadcast distribution in the content preparation system 300. In another example, for the content preparation system 300, ACR fingerprinting may occur in the broadcast chain, where the broadcast feed may be delivered for encoding and uploading. An advantage of performing ACR fingerprinting where the broadcast feed may be delivered for encoding and uploading may be that it enables the content preparation system 300 to combine ACR fingerprinting and encoding and uploading into a single appliance. Another advantage of performing ACR fingerprinting where the broadcast feed may be delivered for encoding and uploading may be that a single broadcast feed is utilized to perform ACR fingerprinting and encoding and uploading. When a single broadcast feed is used, the number of devices and applications to be purchased, installed, monitored, and maintained for ACR fingerprinting and encoding and uploading may be reduced. For example, capital expenditures (CAPEX) and / or operating expenditures (OPEX) may be reduced, such as reduced system facilities (rack space, power, and internet connectivity). Additionally, when a single broadcast feed is used, the number of broadcast feeds generated from the broadcast station's distribution amplifiers can be reduced.

[0070] The content device 302 may receive media content from a content feed and prepare the content to include contextually relevant material that can be delivered to a media device 315. The media device 315 may include an ACR client 325 for performing ACR fingerprint matching locally on the media device. Initially, a request is sent to a server device (e.g., an ACR fingerprint identifier server 105 or a lookup server 206) to identify a video. Once successful, the fingerprint set associated with the video is downloaded to the requesting media device 315. From that point on, most requests for matching can be performed locally on the requesting media device until a low-confidence result is generated (e.g., the fingerprint of the media content no longer matches the downloaded fingerprint set). In response to a failure to match a new fingerprint, a new request can be sent to the server device to match the new fingerprint, which can be matched to a new channel, and the process repeats. Ideally, only one request per video or channel is sent to the server device. However, in some embodiments, more than one request may be made to the server device per video or channel; however, these requests may be fewer than would be made by the media device without local fingerprint matching as described herein.

[0071] In another embodiment, the following pseudo-code represents how ACR fingerprint local matching may be performed on a client device.

[0072]

[0073] In one embodiment, the ACR client uses a local matching algorithm to perform ACR fingerprint matching. It can use the server to match only when necessary, thereby reducing the computation and communication requirements imposed on the server.

[0074] Figure 4 A flow chart illustrating a method 400 for automatic content recognition (ACR) fingerprinting and local matching of a media device according to one embodiment is shown. The method 400 may be performed at least in part by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed by a processing device), firmware, or a combination thereof. The method 400 may be performed by a media device 115 (e.g., Figure 1 In a related embodiment, the ACR client 125 or 325 may execute the method 400. Alternatively, the method 400 may be executed by other processing devices in various types of user devices, portable devices, televisions, projectors, or other media devices.

[0075] refer to Figure 4, method 400 begins with an ACR client sending a request to an ACR server to identify media content based on fingerprints (402). The ACR client may generate a sequence of frames (e.g., two to three seconds worth of frames, depending on the speed at which the content is streamed) as fingerprints that are sent to the server for ACR matching. The server performs server-side matching to identify the media content and sends a set of fingerprints associated with the identified media content back to the ACR client. The ACR client receives the set of fingerprints from the ACR server (404). The ACR client performs a local fingerprint match on the client media device with the set of fingerprints corresponding to the media content (406). The ACR client also determines a match metric (e.g., the quality or confidence level of the match) and compares the match metric against a threshold match level (408). When the match metric is less than the threshold, the ACR client returns to box 402 to send a new request to the ACR server to determine a match with a new channel. However, when the match metric is equal to or greater than the threshold, the ACR client continues to perform local fingerprint matching (406). In this way, the ACR client does not have to continually ping the ACR server to confirm that the media content is being streamed from the same channel or television program.

[0076] Figure 5 A flow chart illustrating a method 500 for local matching of fingerprints for media device fingerprinting of media content according to another embodiment. The method 500 may be performed at least in part by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed by a processing device), firmware, or a combination thereof. The method 500 may be performed by a media device 115 (e.g., Figure 1 In a related embodiment, the ACR client 125 or 325 may execute the method 500. Alternatively, the method 500 may be executed by other processing devices in various types of user devices, portable devices, televisions, projectors, or other media devices.

[0077] refer to Figure 5, the method begins with the media device 115 or 315 generating a first query fingerprint of media content consumed by a user (510). The media device sends the first query fingerprint to the ACR server for content matching (520). The media device receives an identification of the first channel that the user is watching in response to the server finding a match with the first query fingerprint (530). The media device then receives a plurality of fingerprints generated from a location of the media content that is located before where the user is watching the media content (540). The media device stores the plurality of fingerprints received from the server in a local database within a computer-readable medium of the media device (550). The computer-readable storage medium of the media device may be a first-in-first-out (FIFO) memory, where the oldest of the plurality of fingerprints exists first when the memory is filled. In instances where the media content is not a live video stream (e.g., it has been pre-stored), the server may send all fingerprints of the program on the first channel (assuming the media device can store them all).

[0078] The media device then proceeds to match the next query fingerprint with the multiple locally stored fingerprints and, in doing so, determines whether the user is still watching the same channel or program (560). A match may be determined based on satisfying a level of a match metric threshold between the next query fingerprint and any of the multiple fingerprints received from the server and stored in the local database. The match metric threshold may reflect a confidence level that the error of the matching result is low enough so that there is a sufficiently high probability of a match.

[0079] When a match is still found in the multiple fingerprints, the first channel is still being watched and the method loops back to box 540, continuing to receive multiple fingerprints generated from locations of the media content that are in front of where the user is watching (540), and storing the multiple fingerprints in the local database (550) to continue local ACR matching.

[0080] When no match is found, the channel on the media device has been changed and the method returns to block 520, where the unmatched fingerprint is sent to the ACR server to find a match with the second channel that the user has changed to on the media device. The method then continues as described above to receive the identification of the second channel and multiple fingerprints of the media content of the second channel from the server for local storage.

[0081] With further reference to block 540, the server may send not only a plurality of fingerprints of media content of the first channel currently being viewed by the user, but also a second plurality of fingerprints of media content being streamed to channels located adjacent to the first channel on the channel guide. For example, if CBS is the first channel and is located between NBC and CBS, the media device may also receive from the server a plurality of fingerprints of content being streamed to NBC and CBS.

[0082] Alternatively or additionally, the ACR server (optionally in conjunction with Figure 2 The advertising target determiner 212 of the user can track the channels that the user watches the most (e.g., the user's favorite channels and / or programs) and also send multiple fingerprints to those channels on a continuous basis for local ACR matching. In the case of the last two examples (nearby local channels and the user's favorite channels), an upper limit can be set on how many fingerprints are streamed for storage on the media device, depending on the size of the streaming fingerprints, which is constrained by practical limitations of storage space on some media devices (such as smart TVs or applications on handheld devices).

[0083] Figure 6 The diagram illustrates a diagrammatic representation of a machine in the exemplary form of a computer system 600 within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In some alternative embodiments, the machine may be connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. The machine may operate in the capacity of a server or a client device in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smartphone, a web appliance, a server, a network router, a switch, or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specifies actions to be taken by the machine. Further, while a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute an instruction set (or multiple instruction sets) to perform any one or more of the methodologies discussed herein.

[0084] The computer system 600 may correspond to Figure 1 ACR fingerprint reader server 105. The computer system 600 may correspond to Figure 1 The media device 115, or corresponding to Figure 3 The media device 315 or corresponding to Figure 1 and 2 Any of all or a portion of the content manager 122. The computer system 600 may correspond to at least a portion of a cloud-based computer system, such as discussed herein.

[0085] The computer system 600 includes a processing device 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM) or DRAM (RDRAM)), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 618, which communicate with each other via a bus 630.

[0086] Processing device 602 represents one or more general-purpose processing devices (e.g., microprocessors, central processing units, etc.). More specifically, the processing device may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor that implements other instruction sets, or a combination of instruction sets. Processing device 602 may also be one or more special-purpose processing devices (e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc.). In one embodiment, processing device 602 may include one or more processing cores. Processing device 602 may be configured to execute instructions 626 of the mirror logic for performing the operations discussed herein.

[0087] Computer system 600 may further include a network interface device 608 communicatively coupled to a network 620. Computer system 600 may also include a video display unit 610 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse), a signal generating device 616 (e.g., a speaker), or other peripheral devices. Furthermore, computer system 600 may include a graphics processing unit 622, a video processing unit 628, and an audio processing unit 632. In another embodiment, computer system 600 may include a chipset (not illustrated), which refers to an integrated circuit or group of chips designed to work with processing device 602 and control communications between processing device 602 and external devices. For example, the chipset may be a chipset on a motherboard that links processing device 602 to very high-speed devices (e.g., main memory 604 and a graphics controller) and to a lower-speed peripheral bus (e.g., a USB, PCI, or ISA bus) that links processing device 602 to peripheral devices.

[0088] The data storage device 618 may include a computer-readable storage medium 626 having stored thereon instructions 624 embodying any one or more of the methodologies described herein. The instructions 626 may also reside, completely or at least partially, within the main memory 602 and / or within the processing device 600 during execution thereof by the computer system 604, the main memory 604 and the processing device 602 also constituting computer-readable storage media.

[0089] The computer-readable storage medium 624 may also be used to store instructions 626 using logic and / or software libraries containing methods for invoking the above applications. Although the computer-readable storage medium 624 is shown as a single medium in the exemplary embodiment, the term "computer-readable storage medium" or "computer-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache memory and servers) that store one or more sets of instructions. The term "computer-readable storage medium" should also be taken to include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing the machine to perform any one or more of the methodologies of the embodiments of the present invention. Thus, the term "computer-readable storage medium" should be taken to include, but not be limited to, solid-state memory, as well as optical and magnetic media.

[0090] In the above description, numerous details are set forth. However, those skilled in the art who benefit from the present invention will appreciate that the embodiments can be practiced without these specific details. In some instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the description.

[0091] Although embodiments may be described herein with reference to specific integrated circuits (e.g., in computing platforms or microprocessors), other embodiments may be applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of the embodiments described herein may be applied to other types of circuits or semiconductor devices. For example, the disclosed embodiments are not limited to desktop computer systems or Ultrabooks. TM , and can also be used in other devices such as handheld devices, tablet computers, other thin notebook computers, system-on-a-chip (SOC) devices, and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, smartphones, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications typically include microcontrollers, digital signal processors (DSPs), system-on-a-chips, network computers (NetPCs), set-top boxes, network hubs, wide area network (WAN) switches, or any other system that can perform the functions and operations taught below.

[0092] Although embodiments are described herein with reference to processors or processing devices, other embodiments are applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of embodiments of the present invention may be applicable to other types of circuits or semiconductor devices that may benefit from higher pipeline throughput and improved performance. The teachings of embodiments of the present invention are applicable to any processor or machine that performs data manipulation. However, the present invention is not limited to processors or machines that perform 512-bit, 256-bit, 128-bit, 64-bit, 32-bit and / or 16-bit data operations, and may be applicable to any processor and machine in which manipulation or management of data is performed. In addition, the following description provides examples, and the accompanying drawings show various examples for illustrative purposes. However, these examples should not be interpreted as limiting, as these examples are intended only to provide examples of embodiments of the present invention and are not intended to provide an exhaustive list of all possible implementations of embodiments of the present invention.

[0093] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing techniques most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally conceived to be a consistent sequence of operations that leads to a desired result. The operations are those requiring physical manipulation of physical quantities. Typically, but not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. The blocks described herein may be hardware, software, firmware, or a combination thereof.

[0094] It should be remembered, however, that all of these terms and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as is apparent from the above discussion, it should be understood that throughout this description, discussions utilizing terms such as "detect," "initiate," "determine," "continue," "stop," "receive," "record," and the like refer to actions and processes of a computing system or similar electronic computing device that manipulate and transform data represented as physical (e.g., electronic) quantities within the computing system's registers and memories into other data similarly represented as physical quantities within the computing system's memories or registers or other such information storage, transmission, or display devices.

[0095] The words "exemplary" or "exemplary" are used herein to mean serving as an example, instance or illustration. Any aspect or design described herein as "exemplary" or "exemplary" should not necessarily be considered as being preferred or advantageous over other aspects or designs. Rather, it is intended that the words "exemplary" or "exemplary" be used to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise specified or clear from the context, "X includes A or B" is intended to mean any one of the natural inclusive arrangements. That is, if X includes A, X includes B, or X includes both A and B, then "X includes A or B" is satisfied under any of the aforementioned examples. In addition, the articles "a and an" as used in this application and the appended claims should generally be considered to mean "one or more", unless otherwise specified or clear from the context that it is directed to a singular form. In addition, the use of the terms "one embodiment" or "one embodiment" or "an implementation" or "one implementation" throughout the text is not intended to refer to the same embodiment or implementation unless otherwise described. Furthermore, the terms "first," "second," "third," "fourth," etc., as used herein, are intended to be labels for distinguishing between different elements and may not necessarily have an ordinal meaning according to their numerical designations.

[0096] The embodiments described herein may also relate to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the desired purpose, or it may comprise a general-purpose computer that is selectively activated or reconfigured by a computer program stored in the computer. This computer program may be stored in a non-transitory computer-readable storage medium, such as, but not limited to, any type of magnetic disk (including floppy disks, optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic or optical cards, flash memory, or any other type of medium suitable for storing electronic instructions. The term "computer-readable storage medium" shall be construed to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache memory and servers) that stores one or more sets of instructions. The term "computer-readable medium" shall also be construed to include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine and causing the machine to perform any one or more of the methodologies of the embodiments of the present invention. The term "computer-readable storage medium" should accordingly be taken to include, but not be limited to, solid-state memories, optical media, magnetic media, and any medium capable of storing a set of instructions for execution by a machine and causing the machine to perform any one or more of the methodologies of the present invention.

[0097] The algorithms and displays presented herein are not inherently related to any particular computer or other device. Various general-purpose systems may be used with programs according to the teachings herein, or it may prove convenient to construct more specialized equipment to perform the operations. The required structure of various such systems will be apparent from the following description. In addition, the embodiments of the present invention are described without reference to any particular programming language. It will be appreciated that a variety of programming languages ​​may be used to implement the teachings of the embodiments as described herein.

[0098] The above description sets forth numerous specific details (e.g., examples of specific systems, components, methods, etc.) in order to provide a good understanding of several embodiments. However, it will be apparent to those skilled in the art that at least some embodiments may be practiced without these specific details. In other instances, well-known components or methods are not described in detail or are presented in simple block diagram form to avoid unnecessarily obscuring the teachings of the present invention. Therefore, the specific details set forth above are merely exemplary. Specific implementations may differ from these exemplary details and are still contemplated within the scope of embodiments of the present invention.

[0099] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of the embodiments of the present invention should be determined with reference to the appended claims together with the full scope of equivalents to which such claims are entitled.

Claims

1. A media device comprising: at least one processing device; as well as a non-transitory computer-readable medium storing program instructions that, when executed by the at least one processing device, cause the media device to perform operations including the following; generating a first query fingerprint of first media content being consumed by the media device; Sending the first query fingerprint to the server; In response to the server finding a match with the first query fingerprint: (i) receiving, from the server, an identification of a first channel being viewed via the media device; (ii) receiving from the server a first plurality of fingerprints generated from first upcoming media content, the first upcoming media content is to be displayed on the first channel after the first media content; as well as (iii) receiving from the server a second plurality of fingerprints of second upcoming media content to be displayed on a second channel different from the first channel; storing the first plurality of fingerprints and the second plurality of fingerprints in a local database of the media device; generating a second query fingerprint for second media content being consumed by the media device; comparing the second query fingerprint with the first plurality of fingerprints and the second plurality of fingerprints stored in the local database; as well as Based on the comparison of the second query fingerprint, a determination is made as to whether the media device remains set to the first channel or has changed to the second channel.

2. The media device of claim 1 , wherein the operations further comprise: In response to detecting that the second query fingerprint does not match the first and second pluralities of fingerprints, the second query fingerprint is sent to the server to obtain an identification of a third channel different from the first and second channels.

3. The media device of claim 2, wherein the operations further comprise: The identification of the third channel is received from the server in response to the second query fingerprint matching a corresponding fingerprint in a third plurality of fingerprints stored on the server from third media content streamed to the third channel.

4. The media device of claim 2, wherein detecting the mismatch of the second query fingerprint comprises: It is determined that the second query fingerprint fails to satisfy a threshold of a match metric between the second query fingerprint and any of the first and second pluralities of fingerprints, wherein the match metric includes a confidence level of the match.

5. The media device of claim 1 , wherein the operations further comprise: In response to the server finding a match to the first query fingerprint, a third plurality of fingerprints for third upcoming media content to be displayed on a third channel, different from the first channel and the second channel, is received from the server. 6 . The media device of claim 1 , wherein each of the first query fingerprint and the second query fingerprint comprises a respective sequence of frames of media content spanning a predetermined number of seconds.

7. A method comprising: generating, by a media device, a first query fingerprint of first media content being consumed by the media device; Sending the first query fingerprint by the media device to a server; In response to the server finding a match with the first query fingerprint: (i) receiving, from the server, an identification of a first channel being viewed via the media device; and (ii) receiving, from the server, a first plurality of fingerprints generated from first upcoming media content, the first upcoming media content to be displayed on the first channel after the first media content; as well as (iii) receiving from the server a second plurality of fingerprints of second upcoming media content to be displayed on a second channel different from the first channel; storing, by the media device, the first plurality of fingerprints and the second plurality of fingerprints in a local database of the media device; and generating a second query fingerprint for second media content being consumed by the media device; comparing the second query fingerprint with the first plurality of fingerprints and the second plurality of fingerprints stored in the local database; as well as Based on the comparison of the second query fingerprint, a determination is made as to whether the media device remains set to the first channel or has changed to the second channel.

8. The method according to claim 7, further comprising: In response to detecting that the second query fingerprint does not match the first and second pluralities of fingerprints, the second query fingerprint is sent to the server to obtain an identification of a third channel different from the first and second channels.

9. The method according to claim 8, further comprising: The identification of the third channel is received from the server in response to the second query fingerprint matching a corresponding fingerprint in a third plurality of fingerprints stored on the server from third media content streamed to the third channel.

10. The method of claim 8, wherein detecting the mismatch of the second query fingerprint comprises: It is determined that the second query fingerprint fails to satisfy a threshold of a match metric between the second query fingerprint and any of the first and second pluralities of fingerprints, wherein the match metric includes a confidence level of the match.

11. The method according to claim 7, further comprising: In response to the server finding a match to the first query fingerprint, a third plurality of fingerprints for third upcoming media content to be displayed on a third channel, different from the first channel and the second channel, is received from the server.

12. The method of claim 7, wherein the second channel is located sequentially adjacent to the first channel on a channel guide.

13. The method of claim 7, wherein each of the first query fingerprint and the second query fingerprint comprises a respective sequence of frames of media content spanning a predetermined number of seconds.

14. A non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor of a media device, cause the media device to perform operations comprising: generating a first query fingerprint of first media content being consumed by the media device; Sending the first query fingerprint to the server; In response to the server finding a match with the first query fingerprint: (i) receiving, from the server, an identification of a first channel being viewed via the media device; (ii) receiving from the server a first plurality of fingerprints generated from first upcoming media content, the first upcoming media content is to be displayed on the first channel after the first media content; as well as (iii) receiving from the server a second plurality of fingerprints of second upcoming media content to be displayed on a second channel different from the first channel; storing the first plurality of fingerprints and the second plurality of fingerprints in a local database of the media device; generating a second query fingerprint for second media content being consumed by the media device; comparing the second query fingerprint with the first plurality of fingerprints and the second plurality of fingerprints stored in the local database; as well as Based on the comparison of the second query fingerprint, a determination is made as to whether the media device remains set to the first channel or has changed to the second channel.

15. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise: In response to detecting that the second query fingerprint does not match the first and second pluralities of fingerprints, the second query fingerprint is sent to the server to obtain an identification of a third channel different from the first and second channels.

16. The non-transitory computer-readable medium of claim 15, wherein the operations further comprise: The identification of the third channel is received from the server in response to the second query fingerprint matching a corresponding fingerprint in a third plurality of fingerprints stored on the server from third media content streamed to the third channel.

17. The non-transitory computer-readable medium of claim 15, wherein detecting the mismatch of the second query fingerprint comprises: It is determined that the second query fingerprint fails to satisfy a threshold of a match metric between the second query fingerprint and any of the first and second pluralities of fingerprints, wherein the match metric includes a confidence level of the match.

18. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise: In response to the server finding a match to the first query fingerprint, a third plurality of fingerprints for third upcoming media content to be displayed on a third channel, different from the first channel and the second channel, is received from the server.

19. The non-transitory computer-readable medium of claim 14, wherein the second channel is located sequentially adjacent to the first channel on a channel guide.

20. A media device comprising: at least one processing device; as well as a non-transitory computer-readable medium storing program instructions that, when executed by the at least one processing device, cause the media device to perform operations including the following; generating a first query fingerprint of first media content being consumed by the media device; Sending the first query fingerprint to the server; receiving, from the server, a plurality of fingerprints of upcoming media content to be displayed on a second channel different from the first channel, in response to the server finding a match to the first query fingerprint indicating an identity of a first channel being viewed via the media device; storing the received plurality of fingerprints; generating a second query fingerprint for second media content being consumed by the media device; comparing the second query fingerprint to at least the received plurality of fingerprints; determining whether the media device has changed from the first channel to the second channel based on the comparison of the second query fingerprint with at least the received plurality of fingerprints of the upcoming media content to be displayed on the second channel different from the first channel; as well as A plurality of fingerprints of upcoming media content to be displayed on the first channel is received from the server, wherein the plurality of fingerprints of upcoming media content to be displayed on the first channel is a first plurality of fingerprints, and wherein the plurality of fingerprints of upcoming media content to be displayed on the second channel is a second plurality of fingerprints.

21. The media device of claim 20, wherein the operations further comprise: In response to detecting that the second query fingerprint does not match the first and second pluralities of fingerprints, the second query fingerprint is sent to the server to obtain an identification of a third channel different from the first and second channels.

22. The media device of claim 21, wherein the operations further comprise: The identification of the third channel is received from the server in response to the second query fingerprint matching a corresponding fingerprint in a third plurality of fingerprints stored on the server from third media content streamed to the third channel.

23. The media device of claim 21 , wherein detecting the mismatch of the second query fingerprint comprises: It is determined that the second query fingerprint fails to satisfy a threshold of a match metric between the second query fingerprint and any of the first and second pluralities of fingerprints, wherein the match metric includes a confidence level of the match.

24. The media device of claim 20, wherein the operations further comprise: In response to the server finding a match to the first query fingerprint, a third plurality of fingerprints for third upcoming media content to be displayed on a third channel, different from the first channel and the second channel, is received from the server.

25. The media device of claim 20, wherein the second channel is located sequentially adjacent to the first channel on a channel guide.

26. The media device of claim 20, wherein the first query fingerprint and the second query fingerprint are each a respective sequence of frames of media content spanning a predetermined number of seconds.

27. A method comprising: generating, by a media device, a first query fingerprint of first media content being consumed by the media device; sending the first query fingerprint to a server via the media device; receiving, by the media device from the server, a plurality of fingerprints of upcoming media content to be displayed on a second channel different from the first channel, in response to the server finding a match with the first query fingerprint indicating an identity of a first channel being viewed via the media device; storing the received plurality of fingerprints via the media device; generating, by the media device, a second query fingerprint for second media content being consumed by the media device; comparing, by the media device, the second query fingerprint with at least the received plurality of fingerprints; determining, by the media device, whether the media device has changed from the first channel to the second channel based on the comparison of the second query fingerprint with at least the received plurality of fingerprints of the upcoming media content to be displayed on the second channel different from the first channel; as well as A plurality of fingerprints of upcoming media content to be displayed on the first channel is received from the server, wherein the plurality of fingerprints of upcoming media content to be displayed on the first channel is a first plurality of fingerprints, and wherein the plurality of fingerprints of upcoming media content to be displayed on the second channel is a second plurality of fingerprints.

28. The method of claim 27, further comprising: In response to detecting that the second query fingerprint does not match the first and second pluralities of fingerprints, the second query fingerprint is sent to the server to obtain an identification of a third channel different from the first and second channels.

29. The method of claim 28, further comprising: The identification of the third channel is received from the server in response to the second query fingerprint matching a corresponding fingerprint in a third plurality of fingerprints stored on the server from third media content streamed to the third channel.

30. The method of claim 28, wherein detecting the mismatch of the second query fingerprint comprises: It is determined that the second query fingerprint fails to satisfy a threshold of a match metric between the second query fingerprint and any of the first and second pluralities of fingerprints, wherein the match metric includes a confidence level of the match.

31. The method of claim 27, further comprising: In response to the server finding a match to the first query fingerprint, a third plurality of fingerprints for third upcoming media content to be displayed on a third channel, different from the first channel and the second channel, is received from the server.

32. The method of claim 27, wherein the second channel is located sequentially adjacent to the first channel on a channel guide.

33. The method of claim 27, wherein the first query fingerprint and the second query fingerprint are each a respective sequence of frames of media content spanning a predetermined number of seconds.

34. A non-transitory computer-readable medium storing program instructions that, when executed by at least one processor of a media device, cause the media device to perform operations comprising: generating a first query fingerprint of first media content being consumed by the media device; Sending the first query fingerprint to the server; receiving, from the server, a plurality of fingerprints of upcoming media content to be displayed on a second channel different from the first channel, in response to the server finding a match to the first query fingerprint indicating an identity of a first channel being viewed via the media device; storing the received plurality of fingerprints; generating a second query fingerprint for second media content being consumed by the media device; comparing the second query fingerprint to at least the received plurality of fingerprints; determining whether the media device has changed from the first channel to the second channel based on the comparison of the second query fingerprint with at least the received plurality of fingerprints of the upcoming media content to be displayed on the second channel different from the first channel; as well as A plurality of fingerprints of upcoming media content to be displayed on the first channel is received from the server, wherein the plurality of fingerprints of upcoming media content to be displayed on the first channel is a first plurality of fingerprints, and wherein the plurality of fingerprints of upcoming media content to be displayed on the second channel is a second plurality of fingerprints.

35. The non-transitory computer-readable medium of claim 34, wherein the operations further comprise: In response to detecting that the second query fingerprint does not match the first and second pluralities of fingerprints, the second query fingerprint is sent to the server to obtain an identification of a third channel different from the first and second channels.

36. The non-transitory computer-readable medium of claim 35, wherein the operations further comprise: The identification of the third channel is received from the server in response to the second query fingerprint matching a corresponding fingerprint in a third plurality of fingerprints stored on the server from third media content streamed to the third channel.

37. The non-transitory computer-readable medium of claim 34, wherein the operations further comprise: In response to the server finding a match to the first query fingerprint, a third plurality of fingerprints for third upcoming media content to be displayed on a third channel, different from the first channel and the second channel, is received from the server.

38. The non-transitory computer-readable medium of claim 34, wherein the second channel is located sequentially adjacent to the first channel on a channel guide.

39. The non-transitory computer-readable medium of claim 34, wherein the first query fingerprint and the second query fingerprint are each of a respective sequence of frames of media content spanning a predetermined number of seconds.

Images