Systems, methods, and data stores that facilitate content-related actions

Abstract

When a media client receives a media stream for presentation, the media client can receive metadata identifying the media stream in-band with the media stream, such as a media stream identifier encoded as a watermark in the media stream or carried in-band with the media stream in a packet-based transport stream, among other possibilities. And while the media client is processing the media stream for presentation, the media client can then communicate the received metadata out-of-band to the server to enable the server to readily obtain a reference fingerprint representing the media stream and thereby facilitate content-related actions such as dynamic content modification based on the reference fingerprint.

Description

[0001] This application is a divisional application, the original of which is a PCT application with application number PCT / US2020 / 049374, filed on September 4, 2020, and entered the Chinese national phase on March 11, 2022. The Chinese application number is 202080063939.6, entitled "Using in-band metadata as the basis for access reference fingerprints to facilitate content-related actions".

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Patent Application No. 15 / 929,210, filed January 28, 2020, and U.S. Provisional Patent Application No. 62 / 900,037, filed September 13, 2019, the entire contents of which are incorporated herein by reference. Background Technology

[0004] A typical media client operates to receive analog or digital media streams representing media content such as video and / or audio content, and processes the media streams for presentation on a user interface such as a display screen and / or audio speakers by the media client or associated entities. Examples of such media clients may include televisions, computer monitors, projection systems, speakers, headphones, set-top boxes (e.g., cable or satellite TV receivers), digital video recorders, radios, personal computers, mobile communication devices, game consoles, streaming media players, and / or combinations of these other devices.

[0005] For example, a television can receive broadcast streams (e.g., over the air, from a set-top box, via an internet connection, or otherwise) and present the media content of that broadcast stream to a user. As another example, a set-top box can receive broadcast streams from a multi-channel video program distributor (MVPD) and output the media content of that stream to a television and / or audio / video receiver for playback via a high-definition multimedia interface (HDMI) cable or other interface. As yet another example, a digital video or audio recorder can similarly receive broadcast streams but can store them for later broadcast. As yet another example, a speaker or headphones can receive broadcast audio streams from a radio, computer, or other device and present the audio content of that stream to a user. Many other examples are also possible. Summary of the Invention

[0006] This disclosure provides that, when a media client receives a media stream for presentation, the media client can receive in-band metadata identifying the media stream along with the media stream, such as a media stream identifier encoded as a watermark in the media stream, or a media stream identifier carried along with the media stream in a packet-based transport stream. Furthermore, this disclosure provides that, while the media client is processing the media stream for presentation, the media client can subsequently transmit the received metadata out-of-band to a server, enabling the server to easily obtain a reference fingerprint representing the media stream, thereby facilitating content-related actions such as dynamic content modification based on the reference fingerprint.

[0007] Similarly, this disclosure provides that, when a media client is processing a media stream for rendering, the server can receive from the media client an out-of-band signal providing a media stream identifier, which is extracted by the client from in-band metadata received by the client along with the media stream. Furthermore, this disclosure provides that the server can subsequently use the received media stream identifier as a basis to easily obtain a reference fingerprint representing the media stream being rendered, and the server can use the obtained reference fingerprint as a basis to facilitate content-related actions such as dynamic content modification.

[0008] These and other aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description and, where appropriate, referring to the accompanying drawings. It should be understood that the description provided in the Summary Section and below is intended to illustrate the invention by way of example only, and not by way of limitation. Attached Figure Description

[0009] Figure 1 It is a simplified block diagram of an example system that can apply various publicly available principles.

[0010] Figure 2 This is another simplified block diagram of an example system that can apply various publicly available principles.

[0011] Figure 3 This is a flowchart illustrating a method that can be performed according to this disclosure.

[0012] Figure 4 This is another flowchart illustrating a method that can be performed according to this disclosure.

[0013] Figure 5 It is a simplified block diagram of an example computing system that can operate according to this disclosure.

[0014] Figure 6 This is a simplified block diagram of an example media client that can operate according to this disclosure. Detailed Implementation

[0015] Referring to the attached diagram, as described above, Figure 1 This is a simplified block diagram of an example system capable of applying various publicly available principles. However, it should be understood that such and other arrangements and processes described herein can take many other forms. For example, elements and operations can be reordered, distributed, copied, combined, omitted, added, or otherwise modified. Furthermore, it should be understood that the functions described herein, performed by one or more entities, can be implemented by and / or on behalf of these entities through hardware, firmware, and / or software (e.g., by one or more processing units that execute program instructions).

[0016] like Figure 1 As shown, the example system includes one or more media content sources 100 (e.g., broadcasters, web servers, etc.), one or more media content distributors 102 (e.g., MVPD, such as cable TV providers, satellite providers, wireless broadcast providers, web aggregators, etc.), one or more media content receivers 104 (e.g., cable receivers, satellite receivers, wireless broadcast receivers, computers or other streaming media receivers, etc.), and one or more clients or content presentation devices 106 (e.g., TVs or other display devices, speakers or other audio output devices, etc.).

[0017] In an example implementation, media content source 100 can be a national broadcaster, such as ABC, NBC, CBS, FOX, HBO, and CNN; media content distributor 102 can be a local branch in a designated market area (DMA) and / or other local content distributors; and receiver 104 and content presentation device 106 can be located at a customer premises, such as a residential or commercial establishment. With this or other arrangement, content source 100 can transmit media content to content distributor 102 for distribution to receiver 104 at the customer premises, and the content distributor can distribute media content to receiver 104 on discrete channels (e.g., specific frequencies or other defined channels). Each receiver can then respond to user input or one or more other triggers by tuning to a selected channel and outputting media content arriving at the selected channel to content presentation device 106. Content presentation device 106 can receive and render media content (e.g., display, or otherwise present content).

[0018] When a media client, such as receiver 104 and / or content presentation device 106, is processing a linear media stream for presentation (e.g., when the media client is rendering the media stream for presentation, outputting the media stream for presentation, and / or otherwise processing the media stream for presentation), it may be useful in certain situations for the media client to perform special content-related actions (i.e., actions related to the content of the media stream).

[0019] For example, content-related actions may involve initiating an interactive event, such as starting an overlay of content (e.g., overlaying audio and / or video content) that prompts the user to respond in some way, or related to user interaction with a media client or the user interface associated with presenting the media stream content.

[0020] As another example, content-related actions can involve dynamically modifying a portion of the streaming media content. For instance, it might be useful for a media client to dynamically replace advertisements or other replaceable segments of the media stream with replacement ads or other alternative content, so that when content is played to a user, the user receives the replacement ads, or other replacement content or other replaceable segments that replace the originally included ads. As yet another example, it might be useful for a media client to dynamically supplement a portion of the media content using overlay or split-screen content (such as channel identifiers, contextual information, ad content, etc.), so that when content is played to a user, the user receives the supplementary content as well as the originally included content.

[0021] Furthermore, it may be expected that media clients perform content-related actions at specific points in time within the media stream. For example, if a media client wants to dynamically replace existing ads or other replaceable segments of the media stream with replacement ads or other replacement content, it may be expected that the media client places the replacement ads or other replacement content in the media stream once the existing ads or other replaceable segments have started. Similarly, if a media client wants to supplement a specific section of media content with an overlay associated with that section, it may be expected that the media client starts that overlay at the beginning of that section. Other examples are also possible.

[0022] More specifically, the media stream in question may define a sequence of frames, such as video frames and / or audio frames, and it may be expected that the media client will perform content-related actions on a frame-by-frame basis. For example, for dynamic content replacement, it may be expected that the media client will insert a replacement ad or other replacement content to replace the existing ad or other replaceable segment, starting from the first frame of the existing ad or other replaceable segment. And to add overlay to a specific part of the media stream, it may be expected that the media client will begin the overlay at the first frame of that part of the media stream or at the first frame specific to that part. Similar rationale may exist for other content-related actions.

[0023] Without limitation, the remainder of this disclosure will primarily focus on Dynamic Ad Insertion (DAI), in which a media client dynamically replaces the playback of alternative ads in a media stream with the playback of alternative ads, such as ads selected based on user demographics. It should be understood that the principles disclosed can also be applied to other content-related actions.

[0024] In a typical DAI scenario, the media client may include a television and / or set-top box that processes media streams from linear broadcast feeds that define a particular television channel. For example, the media client may be currently receiving a media stream from a broadcaster (e.g., a terrestrial broadcaster, or a cable or satellite MVPD), or may have previously received and stored a media stream for later playback, and alternative advertisements may appear at any point in that media stream. For example, especially (but not limited to) live broadcasts of sporting events, the timing of advertisements appearing in such media streams may be unpredictable. Alternatively, advertisements may appear at specific scheduled times.

[0025] In this scenario, while the media client is processing the media stream for presentation, the media client can communicate out-of-band with the web server (e.g., via a broadband network connection or otherwise separated from the channels received by the client), and the server can work to prepare the media client to execute DAI at appropriate times during the media client's processing of the media stream, such as when the media stream will include replaceable advertisements.

[0026] To facilitate this, the server can first determine which media stream the media client is processing for presentation, and then the server can determine the point in time when the alternative ad will appear in that determined media stream. The server can then instruct the media client to insert the alternative ad for playback, replacing the alternative ad, starting from that determined point in the media stream.

[0027] Figure 2 This is a simplified block diagram of an example system capable of implementing the DAI process.

[0028] Figure 2 At its top is shown an example media distribution path extending from content source 200 to media client 204 via content distribution system 202. For example, but not limited to, content source 200 may be a national broadcaster such as one of the above, content distribution system 202 may be an MVPD of a local branch of a national broadcaster such as a national broadcaster, and media client 204 may be a content presentation device such as a TV and / or a receiver such as a set-top box.

[0029] Using this arrangement, the example media stream flows from content source 200 to content distribution system 202 along the media distribution path, and then from content distribution system 202 to media client 204. Without limitations, this media stream can include, and thus define, a sequence of digital frames of media content as described above, possibly representing a specific channel of the content. Content source 200 can transmit the media stream to content distribution system 202 as a packet-based transport stream by transmitting fragments of the media stream in the payload of transport stream packets to intermediate node 202. Content distribution system 202 can then facilitate processing by the media client for presentation by forwarding the media stream (which may also be a packet-based transport stream or a media stream that may be de-packed and / or transcoded) to media client 204 for processing.

[0030] like Figure 2 As further shown, media client 204 can be interconnected with communication network 206, which can be a packet-switched network such as the Internet. For example, the media client can act as a node on a local area network (LAN) of a customer's premises, where the media client has an assigned Internet Protocol (IP) address on the LAN, and the LAN has an IP address on network 206. The media client can thus participate in data communication with various entities via network 206 while processing media streams on the media client.

[0031] The Automatic Content Recognition (ACR) server 208 (e.g., a combination of servers including ACR services) that can be accessed via network 206 (e.g., at a defined IP address on the network) is shown. Media clients can utilize this server to participate in example out-of-band communications, such as IP signaling via network 206.

[0032] ACR server 208 can be configured to participate in various ACR operations, such as automatically determining the identity of a media stream (e.g., a channel) being processed by a media client, automatically detecting upcoming ad replacement opportunities in the media stream, and coordinating the implementation of DAI by media client 204.

[0033] To facilitate these or other such operations, ACR server 208 may be provided with reference data 210, which may include a digital reference fingerprint of each of various media streams (e.g., each of various known channels) and a digital reference fingerprint of each of various alternative advertisements that may appear in a given media stream. Among other possibilities, this reference data may be provided by content source 200 and content management server 214.

[0034] In an example implementation, a reference fingerprint for each of the various known media streams (e.g., each of the various known channels) can be established by one or more capture servers that can access broadcast channel feeds distributed by one or more content sources.

[0035] For example, when content source 200 outputs each of the various media streams shown in the figure (e.g., each of the various known channels), an ACR engine operating at content source 200, or an ACR engine operating as a capture server otherwise associated with the content source, can generate a digital reference fingerprint representing each such media stream on a frame-by-frame or otherwise ongoing basis. ACR engine 212 can then timestamp each reference fingerprint with a reference timestamp indicating the relative time of the associated frame in the media stream's reference timeline or according to another reference clock (such as the clock running at the content source). As ACR engine 212 generates these timestamped reference fingerprints, it can then provide the reference fingerprints to ACR server 208, thereby associating the reference fingerprint of each media stream with a unique identifier for that media stream (e.g., a channel identifier (CID)). (Alternatively, the generation of reference fingerprints can be performed at content distribution system 202, and other possibilities exist).

[0036] ACR server 208 can therefore receive these timestamped digital reference fingerprints and store them, along with the associated media stream identification information, as part of reference data 210. For example, ACR server 208 can store the reference fingerprints and their associated media stream identification information in a database relationship.

[0037] Furthermore, according to the DAI promotion agreement, the content management server 214 can receive a copy of each of one or more replaceable advertisements, and the content management server 214 can generate a digital fingerprint representing each such replaceable advertisement, and can provide those digital fingerprints to the ACR server 208. The ACR server 208 can also store those digital fingerprints of the replaceable advertisements as part of the reference data 210.

[0038] Without limitations, the example digital fingerprinting process can be applied on a per-video-frame basis and can involve establishing representations of luminance and / or other video features. For example, for a given video frame, the fingerprint generator can programmatically divide the frame into a grid, and the fingerprint generator can measure the luminance of the frame for each grid cell and generate a bit string, where each bit or series of bits represents the luminance of the corresponding grid cell, or represents a weighted difference between the luminances of certain defined pairs of grid cells, etc. Furthermore, the fingerprint generator can continuously apply this process to generate digital fingerprints over time as a fingerprint sequence (e.g., as a fingerprint stream). For example, the fingerprint generator can periodically or on another defined basis apply the process to each frame, each keyframe, where the bit string of each frame defines the digital fingerprint and / or, on a sliding window basis, use a specified hash, a combination or series of such bit strings, or other representative values ​​defining the digital fingerprint. Other digital fingerprinting processes can also be used.

[0039] When a media client processes an ongoing media stream representing a given content channel, it can similarly generate a query digital fingerprint to facilitate Ad Recognition (ACR). This query digital fingerprint represents a frame of the media content in the stream it is processing. The media client can apply the same digital fingerprinting process used to generate reference fingerprints for media streams and replacement ads, allowing for comparison of the query fingerprint with those reference fingerprints for ACR purposes and facilitating Ad Recognition (DAI). Furthermore, the media client can timestamp each such query fingerprint to indicate the time of the frame represented by the query fingerprint according to the media client's clock.

[0040] In the example implementation, the media client may periodically (or otherwise from time to time) transmit the latest set of query fingerprints generated by the media client to the ACR server via network 206, so that the ACR server can identify the media stream that the media client is processing.

[0041] When the ACR server receives these query fingerprints from the media client, it can compare the query fingerprints with media stream reference fingerprints stored in reference data 210 by the ACR server in an attempt to find a match. And when it is found with sufficient certainty that a query fingerprint matches a reference fingerprint associated with a specific media stream (e.g., a specific channel), the ACR server can reasonably infer that it is the media stream being processed by the media client.

[0042] To compare a given query fingerprint with a given reference fingerprint, an ACR server can compare corresponding portions of the fingerprints to each other to determine whether the portions match exactly or within a defined tolerance. For example, the ACR server can calculate the maximum deviation between fingerprints and determine whether the maximum deviation is within a predetermined tolerance. Furthermore, if the fingerprint is binary, this might be a Boolean decision, or it might involve calculating Hamming distance (as a count of mismatches between corresponding bit positions in the fingerprint), and if the fingerprint is a more complex value, such as a decimal value or a vector (e.g., grayscale values ​​for each region of a video frame), this might involve determining the distance between the values ​​or vectors. Many other examples are also possible.

[0043] Once the ACR server identifies the media stream that the client is presenting, the ACR server can determine in various ways when the alternative advertisement will appear in the identified media stream, and the ACR server can then send an instruction to the media client to start inserting the alternative advertisement at that point in time while the media client is processing the media stream.

[0044] In one example, the ACR server can determine a reference time when an alternative ad will appear in the media stream, and the server can provide that reference time to the media client.

[0045] As an example, the ACR server can access data specifying the scheduled broadcast time of an alternative advertisement within a determined media stream, such as the time when the media stream distributor is scheduled to include the alternative advertisement in the media stream. This scheduled time can be defined in the channel's reference timeline or based on another reference clock, such as the clock running at the distributor. Once the ACR server identifies the media stream that the media client is rendering, the server can refer to this data to determine the scheduled broadcast time of the alternative advertisement. Prior to this time, the ACR server can participate in out-of-band signaling to the media client to provide the media client with an indication of the scheduled broadcast time of the alternative advertisement, and an instruction to begin rendering the alternative advertisement at that upcoming time.

[0046] Alternatively, the ACR server can apply fingerprint-based ACR as a basis to detect the presence of replaceable ads in the media stream in real time, i.e., while the media stream is being routed to the media client. The ACR server can then determine the reference timestamp of the start of the detected replaceable ad and can send an instruction to the media client to display the replaceable ad that started at that point in the media stream.

[0047] This process can utilize the time delay that may exist between the point in time when the content is output from the content source to the media client and the point in time when the media client receives and / or otherwise processes the content of the media stream. Depending on the system configuration, this delay can be approximately 5-10 seconds and can vary between clients. Given this delay, if the ACR server communicates with the media client out-of-band via a relatively high-speed network connection, such as network 206, the ACR server can give the media client sufficient advance notice of upcoming alternative advertisements in the media stream, allowing the media client to prepare to present the alternative advertisements when they appear in the media stream being processed by the media client.

[0048] In one implementation of this example, for instance, once the ACR server identifies the media stream being presented by the media client, the ACR server can begin comparing a reference fingerprint of that media stream with fingerprints of various replaceable advertisements. And when it is sufficiently determined that the identified reference fingerprint of the media stream matches the fingerprint of a specific replaceable advertisement, the ACR server can infer that the replaceable advertisement exists in the media stream being processed by the media client. The ACR server can then determine a reference time in the media stream from the reference fingerprint timestamp when the first frame of the replaceable advertisement appears in the media stream, and the ACR server can transmit a marker of that time to the media client, which has an indication for the media client to begin inserting a replaceable advertisement in place of the replaceable advertisement at that specific time.

[0049] One problem with using reference time in these examples is that the media client's timeline for rendering the media stream may not match the media stream's reference timeline. As a result, the reference time provided by the ACR server to the media client may not correctly inform the media client when the alternative ads will appear in the media client's media stream.

[0050] One basis for this time difference may be the aforementioned time delay, which is the time delay between the point in time when the content source outputs the media stream for distribution and the point in time when the media client receives and / or otherwise processes the content of that media stream. If the reference time provided by the ACR server represents the start time of the alternative advertisement from the broadcaster's perspective, the time when the alternative advertisement actually appears in the media stream on the media client will be offset from that reference time by broadcast delay.

[0051] Alternatively or additionally, another basis for the time difference may be whether the media client is presenting the media stream in a time-shifted manner, such as when the media client previously received and stored a linear feed of the media stream and now retrieves the feed from storage and presents it. In this case, the media client's timeline for presenting the media stream will be shifted from the broadcast time of the media stream by at least the duration of that time shift.

[0052] To help overcome this technical challenge, ACR servers and / or media clients can work to establish a time mapping or "synchronization lock" between a reference time and client time. Specifically, the ACR server and / or media client can calculate an offset between (i) the reference time (e.g., the broadcast time of the media stream's content or time defined along a timeline of the channel itself) and (ii) the client time (e.g., the time when the media client renders the corresponding content of the media stream). The ACR server and / or media client can then apply this time offset as the basis for conversions between the reference time and client time, such as converting the reference time of the start of an alternative advertisement in the media stream to the client time when the media client processes the media stream.

[0053] ACR and / or media clients can establish this synchronization lock in various ways.

[0054] As an example, an ACR server can determine the time offset by comparing the timestamps of one or more instances of a media stream's reference fingerprint with the timestamps of a query fingerprint that match the media stream's reference fingerprint. Specifically, once the ACR server has determined the media stream being presented by the media client, or during the process of that determination, the ACR server can perform fingerprint matching to definitively find one or more instances of reference fingerprints that match the query fingerprint provided by the client. For each such match, the ACR server can calculate the difference between the reference fingerprint timestamp and the query fingerprint timestamp. The ACR server can assume that the time offset is the difference calculated over a given match or as an average of multiple such match points.

[0055] Given the calculated time offset, the ACR server can then convert the reference time for the start of the replaceable ad in the media stream to the corresponding client time, and thus convey this client time to the media client as an indication of when the replaceable ad starts and, consequently, when the media client should begin displaying the replaceable ad. Alternatively, the ACR server can transmit the calculated time offset to the media client, and the media client can then apply this time offset to convert the server-provided reference time for the start of the replaceable ad to the corresponding client time for the start of the replaceable ad, so that the media client can then begin inserting the replaceable ad at that time.

[0056] Alternatively, once the ACR server has identified the media stream being processed by the media client, it can provide the media client with a set of timestamped reference fingerprints of the media stream, and the media client can perform client-side fingerprint matching to determine the time offset. That is, the media client can perform fingerprint matching to definitively find one or more instances of reference fingerprints that match the query fingerprint generated by the client. For each such match, the media client can calculate the difference between the reference fingerprint timestamp and the client fingerprint timestamp. The media client can then assume that the time offset is the difference calculated over a given match or the average of multiple such matches.

[0057] In this example, if the media client receives a reference time from the ACR server indicating the start time of a replaceable ad in the media stream, the media client can convert that reference time to the corresponding client time, allowing the media client to subsequently display the replaceable ad at that client time. Alternatively, the media client can send a calculated time offset to the ACR server, which can then apply that time offset to the reference time of the start of the replaceable ad in the media stream to determine the corresponding client time. The ACR server can then provide the determined client time to the media client, allowing the media client to insert the replaceable ad at that client time.

[0058] In an alternative implementation, the ACR server can use fingerprint-based ACR to detect the presence of replaceable ads in the media stream in real time as the media stream is being routed to the media client, and the ACR server can then provide the media client with a set of associated reference fingerprints so that the media client can perform client-side matching to detect when replaceable ads will appear in the media stream, and thus when the media client should begin inserting replaceable ads.

[0059] That is, once the ACR server has identified a media stream being processed by a media client, the ACR server can then compare the reference fingerprint of the identified media stream with predetermined fingerprints of various alternative advertisements, and thereby detect the presence of a specific alternative advertisement in the media stream as described above. After detecting an alternative advertisement in the media stream, the ACR server can then obtain a set of reference fingerprints representing frames of the media stream, which indicate the start of an alternative advertisement in the media stream, and include at least that start of the alternative advertisement. The ACR server can then transmit an indication of which reference fingerprints and which reference fingerprint indicates the start of the alternative advertisement to the media client. In response, the media client can then perform client fingerprint matching to detect when an alternative advertisement will appear in the media stream being processed by the media client. The media client can then accordingly begin inserting the alternative advertisement at the detected point in time.

[0060] The process may also utilize the aforementioned time delay, which may exist between the time when the distributor outputs the content of the media stream for distribution and the time when the media client receives and / or otherwise processes the content of the media stream.

[0061] Here, if the ACR server communicates with the media client out of band via a relatively high-speed network connection, the ACR server can provide the media client with a sufficient reference fingerprint before the replaceable advertisement appears in the media stream of the media client, so that the media client can apply client-side matching and prepare to start replacing advertisements on a frame-by-frame basis.

[0062] In an example implementation of DAI, the ACR server may also provide the media client with additional information to facilitate ad replacement. For example, the ACR server may provide the media client with an address (e.g., a Uniform Resource Locator (URL)) or other information that enables the media client to obtain replacement ads that should be inserted by the media client from the content replacement server 216, from the media client's local data storage device, or from another location, and provides duration and other information to enable the media client to obtain appropriate replacement ads for that purpose. The media client can thus obtain replacement ads and can accordingly insert replacement ads in place of replacement ads while the media client is processing an ongoing media stream for presentation.

[0063] Unfortunately, a potential technical problem in the above process is that the ACR server may spend more time than expected or actually required to initially determine which media stream the media client is processing via fingerprint-based ACR, i.e., to initially identify the media stream the media client is processing. Therefore, the ACR server may spend more time than expected or actually required to determine which reference fingerprints can be used for the various purposes described above, such as (i) establishing a synchronization lock to determine when the media client should begin inserting replacement ads, or (ii) performing server-side or client-side fingerprint-based ACR to detect the presence of replacement ads in the media stream the media client is processing, and other possibilities.

[0064] For example, the ACR server searching reference data 210 to definitively find a media stream reference fingerprint that matches the query fingerprint provided by the client may take longer than expected or actually required. Furthermore, in some cases, the ACR server may definitively find that the query fingerprint provided by the client matches reference fingerprints representing each of multiple different media streams (as a multi-match scenario). In this case, the ACR server may then need to engage in additional time-consuming processing to determine which of those multiple media streams is the one being presented by the media client.

[0065] Unfortunately, if the ACR server takes too long to determine which media stream a media client is processing, the ACR may not be able to prepare the media client in time for a given ad replacement opportunity (DAI). Furthermore, the same problem may exist for other types of content-related actions.

[0066] This disclosure provides a useful mechanism that can help overcome this technical problem. According to this disclosure, as described above, when a media client is processing a media stream, such as a channel, to be presented (in real time or using time shift), the media client receives in-band metadata identifying the media stream, and based on the received in-band metadata, sends a representation of the metadata to an ACR server, etc., via out-of-band communication, so that the server can easily determine the media stream being processed by the media client and take the associated operations as described above.

[0067] In a representative implementation, the action of a media client receiving in-band metadata involves the media client receiving metadata (e.g., simultaneously or concurrently with the media client receiving the media stream), rather than the media client receiving metadata being separate from the media client receiving the media stream.

[0068] For example, in-band metadata can be metadata encoded into the media stream itself. For instance, using any of the various watermarking techniques now known or developed in the future, in-band metadata can be steganographically encoded as a watermark (e.g., an audio or video watermark) within the media stream. In this case, the media client can evaluate the media stream to detect the presence of the watermark, and upon detection, the media client can decode the watermark and thus read the metadata from the media stream. Alternatively, the metadata can be encoded in the media stream in other ways, such as within the vertical auxiliary data (VANC) space of the video frame, in which case the media client can read the metadata from the VANC space of the video frame, among other possibilities.

[0069] As another example, if the media stream is carried in a transport stream (e.g., in a packet stream), the transport stream may carry metadata within the header of each of one or more transport stream packets and / or within one or more specific packets in the transport stream. In this case, the media client can evaluate the transport stream to detect the presence of metadata, and then read the metadata from the transport stream.

[0070] These or other forms of in-band metadata can be provided using a variety of protocols, such as, but not limited to, those defined by standards organizations such as the Society of Cable Telecommunications Engineers (SCTE) (e.g., SCTE104 or SCTE35), the Advanced Television Systems Committee (ATSC) (e.g., ATSC3.0), or the European Telecommunications Standards Institute (e.g., Hybrid Broadcast Broadband (HBB) TV (TS102796)).

[0071] As is known in the art, watermarking involves permanently embedding or otherwise encoding information into media content, enabling the recipient of the media content to decode and extract the information, but this may be imperceptible to the user presenting the media content. This is the opposite of inserting such information into packet headers, etc., without encoding the information into the underlying media content itself. Watermarking permanently alters the media content and may be impossible to remove.

[0072] More specifically, watermarking media content can involve encoding code that can be mapped to associated information into the media content, or more directly encoding the associated information into the media content. In example implementations, the watermark code can be on the order of 24 bits, and watermarking can be done in the audio and / or video components of the media content, depending on the format of the media content.

[0073] Existing audio watermarking techniques include, but are not limited to, inserting audio energy into an audio signal or otherwise modifying one or more features of the audio signal in a way that can be detected by a properly configured watermark detection algorithm but is masked and inaudible (e.g., audio features and / or encoding methods are sufficient to mask the watermark from being detected by humans) – i.e., shorthand or psychoacoustic coding.

[0074] U.S. Patent No. 8,359,205 (published January 22, 2013, entitled "Methods and Apparatus to Perform Audio Watermarking and Watermark Detection and Extraction"), U.S. Patent No. 8,369,972 (published February 5, 2013, entitled "Methods and Apparatus to Perform Audio Watermarking Detection and Extraction"), U.S. Patent Publication No. 2010 / 0223062 (published September 2, 2010, entitled "Methods and Apparatus to Perform Audio Watermarking and Watermark Detection and Extraction"), U.S. Patent No. 6,871,180 (published March 22, 2005, entitled "Decoding of Information in Audio Signals"), and U.S. Patent No. 5,764,763 (published June 9, 1998, entitled "Apparatus and Methods for Including Codes in Audio Signals and...") Decoding), U.S. Patent No. 5,574,962 (published November 12, 1996, entitled "Method and Apparatus for Automatically Identifying a Program Including a Sound Signal"), U.S. Patent No. 5,581,800 (published December 3, 1996, entitled "Method and Apparatus for Automatically Identifying a Program Including a Sound Signal"), U.S. Patent No. 5,787,334 (published July 28, 1998, entitled "Method and Apparatus for Automatically Identifying a Program Including a Sound Signal"), and U.S. Patent No. 5,450,Document No. 490 (published on September 12, 1995, entitled "Apparatus and Methods for Including Codes in Audio Signals and Decoding") describes examples of audio watermarking and corresponding watermark detection techniques; their entire contents are incorporated herein by reference.

[0075] On the other hand, existing video watermarking technologies involve embedding code into the video components of media content in a way that can be detected by properly configured watermark detection algorithms but is masked by human visual detection.

[0076] Examples of video watermarking techniques include various spatial domain techniques, such as flipping pixels, embedding the watermark into the least significant bit, and adding pseudo-random noise maps to the video, as well as various frequency domain techniques, such as SVD domain watermarking, discrete Fourier transform watermarking, discrete cosine transform watermarking, discrete wavelet transform watermarking, and principal component analysis watermarking. Other examples are also possible.

[0077] Any of these or other watermarking techniques may be used for this purpose, or in a manner that facilitates the use of currently disclosed features for watermarking.

[0078] In example implementations, the content source, content distributor, or other entity involved in providing the media stream that will ultimately be processed by the media client can provide a unique identifier for the media stream as in-band metadata, by watermarking the media stream identifier into the media stream itself, by writing the media stream identifier as transport stream metadata into a transport stream that also carries the media stream, and / or in another manner. Furthermore, the way in-band metadata is provided can be changed as the media stream is routed to the media client. For example, at some point in the media distribution path, as the media stream flows towards the media client, the transport stream metadata can be converted to a watermark, among other possibilities.

[0079] To enable the ACR server to quickly associate the in-band media stream identifier with a reference fingerprint representing the media stream, the media stream identifier can be the same identifier specified for the media stream in reference data 210. For example, if the media stream represents a specific channel with a specific CID, the media stream identifier provided as in-band metadata can be that same CID. Alternatively, the in-band media stream identifier can be code or other data that can be easily mapped to the same media stream identifier associated with the reference fingerprint, for example, through a database lookup.

[0080] The entity providing the media stream identifier as in-band metadata can also repeat this process throughout the media stream (e.g., along the duration of the media stream) to allow media clients to easily detect and decode the watermark and obtain the media stream identifier. For example, as the media stream progresses, the entity can periodically or otherwise watermark the media stream identifier into the media stream or add the media stream identifier as transport stream metadata to the transport stream carrying the media stream. Furthermore, the entity can accomplish this using predetermined placement within the media stream or transport stream and / or by utilizing predetermined data structures or one or more other predetermined attributes, allowing media clients to discover the presence of the in-band metadata by searching for these predetermined attributes.

[0081] While the media client is processing the media stream for presentation, it can thus discover and read in-band metadata to easily obtain the media stream's identifier. In response, the media client can then easily transmit the obtained media stream identifier to the ACR server via out-of-band signaling, enabling the ACR server to easily obtain the associated reference fingerprint to facilitate useful content-related actions.

[0082] In the example implementation, to account for possible channel changes, the media client may repeat this operation for each in-band instance of the media stream identifier discovered by the media client during media streaming. Alternatively, the media client may not repeat the transmission to the ACR server unless and until the in-band media stream identifier changes or a predetermined time period has elapsed.

[0083] The out-of-band transmission of the media stream identifier obtained by the media client from in-band metadata to the ACR server may involve the media client transmitting the media stream identifier to the IP address of the ACR server in a packet-based communication manner according to any agreed protocol on network 206. Alternatively, the media client may therefore receive the media stream identifier as part of the in-band metadata accompanying the media stream, which is transmitted from the content delivery system 202 via the media distribution path, and then the media client may transmit the obtained media stream identifier out-of-band to the ACR server via network 206.

[0084] Upon receiving a media channel identifier from in-band metadata obtained by the media client and transmitted out-of-band to the ACR server, the ACR server can then easily obtain a reference fingerprint representing that specific, identified media stream. For example, the ACR server can query reference data 210 based on the media stream identifier to easily obtain a reference fingerprint associated with the media stream identifier and thus representing the identified media stream. Furthermore, based on the media stream identifier, the ACR server can obtain additional reference fingerprints representing the identified media stream.

[0085] The ACR server can then readily utilize the obtained reference fingerprint in any of the foregoing methods, and other possibilities, to facilitate content-related actions. For example, the ACR server can use the obtained reference fingerprint as (i) to establish a synchronization lock to facilitate determining when a media client should begin rendering alternative advertisements, and / or (ii) as a basis for server-side or client-side fingerprint-based ACR detection of the presence of alternative advertisements in a media stream that a media client is processing for rendering, and other possibilities.

[0086] Conveniently, this process thus allows the ACR server to obtain a reference fingerprint representing the media stream being processed by the media client, without requiring the ACR server to perform query / reference fingerprint matching to determine which reference fingerprints match the query fingerprint generated by the client representing the media stream. In other words, alternative implementations may additionally involve matching query fingerprints with reference fingerprints for one reason or another.

[0087] Figure 3 This is a flowchart illustrating an example method that can be performed according to this disclosure.

[0088] like Figure 3 As shown, at box 300, the method includes a media client receiving a media stream (e.g., a television channel) and the media client receiving an identifier of the media stream as in-band metadata accompanying the media stream. Furthermore, at box 302, which can occur in parallel with box 300, the method includes a media client processing a media stream for presentation. Additionally, at box 304, the method includes, while the media client is processing the media stream for presentation, the media client transmitting, via out-of-band signaling, the identifier of the media stream received by the media client as in-band metadata accompanying the media stream to the server, so that the server can obtain a reference fingerprint representing the media stream without requiring the server to search for a reference fingerprint that matches a query fingerprint representing the media stream being processed by the media client. Furthermore, at box 306, the method includes the media client receiving an instruction from the server for the media client to perform a content-related action at a specific time within the media stream, the specific time being established based on the obtained reference fingerprint.

[0089] Based on the above discussion, the content-related actions in this method can involve dynamic content revisions, such as DAI, as well as other possibilities.

[0090] Furthermore, as described above, the act of a media client receiving an identifier as in-band metadata accompanying the media stream can involve the media client receiving an identifier that is steganographically encoded as a watermark in the media stream and / or the media client receiving an identifier carried in a transport stream that carries the media stream to the media client. As described above, out-of-band signaling can involve signaling on a broadband network connection or other optimal high-speed communication interface.

[0091] Furthermore, as described above, the media client may include a content presentation device. In this case, processing the media stream for presentation may include the media client presenting the content of the media stream on a user interface and / or otherwise processing the content of the media stream for such presentation. Alternatively or additionally, the media client may include another device, such as a set-top box or other receiver, that interacts with the content presentation device or other associated device. In this case, processing the media stream for presentation may include outputting the media stream for presentation by the associated device and / or otherwise processing the content of the media stream to facilitate content presentation.

[0092] In addition, the actions taken by a media client to process the media stream for presentation may also include the actions of receiving the media stream, as well as other possibilities.

[0093] As discussed above, actions that enable the server to obtain a reference fingerprint representing the media stream can enable the establishment of a synchronization lock that maps the reference time to the client time. Furthermore, the establishment of the synchronization lock allows the conversion of the reference time of content-related actions into the client time of those actions, thereby facilitating the execution of content-related actions at the desired time within the media stream.

[0094] Alternatively or additionally, actions that enable the server to obtain a reference fingerprint representing the media stream can enable the server to perform a fingerprint comparison between a replaceable content fragment (e.g., a replaceable advertisement) and the obtained reference fingerprint to detect the presence of the replaceable content fragment in the media stream, which can facilitate dynamic content revisions, among other possibilities.

[0095] The various other features mentioned above can also be applied to this situation, and vice versa.

[0096] Figure 4 This is another flowchart illustrating a method that can be performed according to this disclosure.

[0097] like Figure 4As shown, at box 400, the method includes, while a media client is processing a media stream for presentation, the server receiving an out-of-band signal from the media client providing the server with an identifier for the media stream, which is extracted from in-band metadata received by the media client from the media client along with the media stream. At box 402, the method then includes the server using the received media stream identifier as a basis to obtain a reference fingerprint representing the media stream, without the server searching for a reference fingerprint that matches a query fingerprint representing the media stream being processed by the media client. Furthermore, at box 404, the method includes the server using the obtained reference fingerprint as a basis to determine a point in the media stream where the media client should perform a content-related action. At box 406, the method includes the server instructing the media client to perform a content-related action at the determined point.

[0098] The features discussed above can also be applied to this context, and vice versa. Without limitation, for example, content-related actions can include DAI or other dynamic content revisions.

[0099] Figure 5 This is a simplified block diagram of an example computing system capable of operating according to this disclosure. The computing system may represent a server, such as the ACR server 208 discussed above, and others.

[0100] like Figure 5 As shown, the computing system includes a network communication interface 500, a processing unit 502, and a non-transitory data storage 504, any one or all of which can be integrated together, or as shown, communicatively connected together via a system bus, network, or other connection mechanism 506.

[0101] Network communication interface 500 may include one or more network connection mechanisms to facilitate communication with one or more other entities on a network such as network 206. Each such network communication interface may include a wireless or wired Ethernet interface or other types of network interface for IP communication and / or other types of network communication.

[0102] Processing unit 502 may include one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., application-specific integrated circuits). Non-transitory data memory 504 may include one or more volatile and / or non-volatile storage components, such as optical memory, magnetic memory, or flash memory. Furthermore, as shown, the data memory 504 of the example computing system stores or otherwise embeds program instructions 508. These program instructions can be executed by processing unit 502 to perform (e.g., cause the computing system to perform) the various operations described herein. For example, if the computing system represents ACR server 208, the instructions may be executable to perform the various ACR server operations discussed above.

[0103] The above features can also be achieved in this context, and vice versa.

[0104] at last, Figure 6 This is a simplified block diagram of an example media client operable according to this disclosure. Consistent with the discussion above, the media device can take various forms, such as content presentation device 106 and / or receiver 104, and it can also be a combination of multiple such devices, possibly a content presentation device with an integrated receiver, and other possibilities.

[0105] like Figure 6 As shown, the example media client includes a content communication interface 600, a content presentation interface 602, a network communication interface 604, a processing unit 606, and a non-transitory data storage 608. Any one or all of them can be integrated together, or as shown, they can be communicatively connected together via a system bus, network, or other connection mechanism 610.

[0106] Content communication interface 600 may include a physical communication interface for receiving and / or outputting media content such as media streams. Thus, the content communication interface may include one or more wired and / or wireless interfaces for establishing communication with analog or digital media content via a media distribution path and for receiving analog or digital media content, and in some embodiments, for outputting the received media content in analog or digital form for reception by an associated entity. Therefore, the content communication interface may include interfaces for receiving media content from a content distributor, such as coaxial cable connections, antennas, receivers, tuners, etc., and a local communication interface for outputting media content to an associated device, the local communication interface conforming to protocols such as DVI, HDMI, VGA, USB, Bluetooth, WIFI, etc. Other examples are also possible.

[0107] The content presentation interface 602, which may be included in an end-user content presentation device, may include one or more components to facilitate the presentation of received media content. As an example, the content presentation interface 602 may include a user interface such as a display screen and / or speakers, and one or more drivers or other components for processing the received media content to facilitate its presentation on the user interface.

[0108] Network communication interface 604 may include network connectivity mechanisms to facilitate communication on a network such as network 206, and / or for direct or networked communication with one or more other local or remote entities. Thus, the network communication interface may include a wireless or wired Ethernet interface or other types of network interfaces for IP communication and / or other types of network communication.

[0109] The processing unit 606 may then include one or more general-purpose processors (e.g., microprocessors) and / or one or more special-purpose processors (e.g., application-specific integrated circuits). The non-transitory data memory 608 may include one or more volatile and / or non-volatile storage components, such as optical memory, magnetic memory, or flash memory. Furthermore, the data memory 608 stores program instructions 612 that can be executed by the processing unit 606 to perform (e.g., cause a media client to perform) the various operations described herein.

[0110] For example, these operations may include receiving a media stream via a content communication interface, and receiving an identifier of the media stream as in-band metadata accompanying the media stream via the content communication interface. Furthermore, the operations may include processing a media stream for presentation. This operation may include, when processing the media stream for presentation, transmitting the identifier of the media stream received as in-band metadata accompanying the media stream to the server via a network communication interface in out-of-band signaling, so that the server can obtain a reference fingerprint representing the media stream without requiring the server to search for a reference fingerprint that matches a query fingerprint representing the media stream being processed for presentation. The operation may include receiving an instruction from the server via the network communication interface for a media client to perform a content-related action at a specific time within the media stream, the specific time being established based on the obtained reference fingerprint.

[0111] The above features can also be achieved in this context, and vice versa.

[0112] In cases where this disclosure relates to individual devices, servers, modules, or other entities, this disclosure also considers the possibility that each such entity is a single entity or a combination of multiple entities. For example, a given entity may include a platform or system comprising multiple devices configured to interact together or to operate independently or otherwise.

[0113] This disclosure also contemplates one or more computer-readable media that have, store, or otherwise embed program instructions executable by a processing unit (e.g., one or more processors) to perform the various described operations.

[0114] Exemplary embodiments have been described above. However, those skilled in the art will understand that changes and modifications can be made to these embodiments without departing from the true scope and spirit of the invention.

Claims

1. A system that facilitates content-related actions, comprising: a processing unit; a non-transitory data store; and program instructions stored in the non-transitory data store and executable by the processing unit for performing operations comprising: receiving, from a media client, out-of-band signaling while the media client is processing a media stream for presentation, the out-of-band signaling providing an identifier of the media stream, the identifier of the media stream being extracted by the media client from in-band metadata received by the media client with the media stream, using the received identifier of the media stream as a basis to obtain a reference fingerprint representing the media stream based on a lookup for the identifier without searching for a reference fingerprint that matches a query fingerprint representing the media stream being processed by the media client, wherein using the received identifier as a basis to obtain a reference fingerprint based on a lookup for the identifier comprises referencing predetermined reference data that associates various identifiers with respective reference fingerprints to determine, based on the reference data, a reference fingerprint associated with the identifier, using the obtained reference fingerprint as a basis to determine a point in the media stream at which the media client should perform a content-related action, and causing the media client to perform the content-related action at the determined point. The content-related action comprises dynamic ad insertion.

2. The system of claim 1, wherein, The identifier of the media stream extracted by the media client from in-band metadata received by the media client with the media stream comprises an identifier extracted by the media client from a watermark in the media stream.

3. The system of claim 1, wherein, The identifier of the media stream extracted by the media client from in-band metadata received by the media client with the media stream comprises an identifier extracted by the media client from a transport stream that carries the media stream to the media client.

4. The system of claim 1, wherein, Processing the media stream for presentation comprises at least one of presenting content of the media stream on a user interface or outputting the media stream for presentation through an associated device.

5. The system of claim 1, wherein, Causation of the media client to perform the content-related action at the determined point comprises outputting an indication for communication to the media client, the indication for causing the media client to perform the content-related action at the determined point in the media stream.

6. The system of claim 1, wherein, The operations further comprise:

7. The system of claim 1, wherein, using the obtained reference fingerprint as a basis to establish a synchronization lock that maps reference times to client times, wherein the establishment of the synchronization lock enables conversion of reference times of content-related actions to client times of content-related actions. The operations further comprise:

8. The system of claim 1, wherein, performing a fingerprint comparison between an alternative content segment and the obtained reference fingerprint to detect that the alternative content segment is present in the media stream.

9. A non-transitory data store that facilitates content-related actions, containing program instructions executable by one or more processors for causing a media client to perform operations comprising: receiving a media stream and receiving an identifier of the media stream as in-band metadata with the media stream; ​ processing a media stream for presentation; and transmitting, by out-of-band signaling, to a server, an identifier of the media stream received as in-band metadata with the media stream, when processing the media stream for presentation, to enable the server to obtain a reference fingerprint representing the media stream based on a lookup for the identifier, without the server searching for a reference fingerprint matching a query fingerprint representing the media stream being processed for presentation, wherein obtaining the reference fingerprint based on the lookup for the identifier comprises referencing predetermined reference data associating various identifiers with respective reference fingerprints to determine, based on the reference data, a reference fingerprint associated with the identifier.

10. The non-transitory data store of claim 9, wherein, The operations further include: receiving an indication from the server to perform a content-related action at a particular time in the media stream, wherein the particular time is established based on the obtained reference fingerprint.

11. The non-transitory data store of claim 10, wherein, The content-related action includes dynamic advertisement insertion.

12. The non-transitory data store of claim 10, wherein, Enabling the server to obtain the reference fingerprint representing the media stream enables establishing a synchronization lock mapping a reference time to a client time, wherein the establishment of the synchronization lock enables converting a reference time of the content-related action to a client time of the content-related action.

13. The non-transitory data store of claim 9, wherein, Enabling the server to obtain the reference fingerprint representing the media stream enables the server to perform a fingerprint comparison between a replaceable content segment and the obtained reference fingerprint to detect that the replaceable content segment is present in the media stream.

14. The non-transitory data store of claim 9, wherein, Receiving the identifier as in-band metadata with the media stream includes receiving the identifier steganographically encoded as a watermark in the media stream or receiving the identifier carried in a transport stream carrying the media stream.

15. The non-transitory data store of claim 9, wherein, Processing the media stream for presentation includes at least one of presenting content of the media stream on a user interface or outputting the media stream for presentation by an associated device.

16. A method of facilitating a content-related action, comprising: receiving, by a media client, a media stream and receiving, by the media client, an identifier of the media stream as in-band metadata with the media stream; processing, by the media client, the media stream for presentation; and transmitting, by the media client, to a server, the identifier of the media stream received by the media client as in-band metadata with the media stream, when processing the media stream for presentation, to enable the server to obtain a reference fingerprint representing the media stream based on a lookup for the identifier, without the server searching for a reference fingerprint matching a query fingerprint representing the media stream being processed by the media client.

Images