Content modification system with transmission delay-based features

Abstract

In an aspect, a method includes determining a content transmission latency between a content distribution system and a content presentation device. The method also includes selecting, using at least the determined content transmission latency as a basis, a reference fingerprint dataset from a plurality of reference fingerprint datasets that corresponds to the determined content transmission latency. The method also includes sending the selected reference fingerprint dataset that corresponds to the determined content transmission latency to the content presentation device to facilitate detection by the content presentation device of a match between query fingerprint data representative of content received by the content presentation device and at least a portion of reference fingerprint data in the sent reference fingerprint dataset.

Description

[0001] This application is a divisional application. Its original application is PCT application number PCT / US2020 / 032118, filed on May 8, 2020, which entered the Chinese national phase on January 10, 2022. The Chinese application number is 202080050171.9, entitled "Content Modification System with Transmission Delay-Based Features".

[0002] Cross-reference to related applications

[0003] This disclosure claims priority to U.S. Patent Application No. 16 / 836,082, filed March 31, 2020; U.S. Patent Application No. 16 / 836,098, filed March 31, 2020; and U.S. Provisional Patent Application No. 62 / 846,405, filed May 10, 2019, all of which are incorporated herein by reference in their entirety. Technical Field

[0004] This application relates to the technical field of methods and systems with characteristics based on transmission delay. Background Technology

[0005] To deliver and present content to end users, content providers can send content to one or more content distribution systems, each of which can then send the content to one or more corresponding content presentation devices to output the content and present it to the relevant end users. This hierarchical arrangement facilitates convenient and widespread content distribution.

[0006] For example, to deliver video content from a video content provider to end users across the United States, the provider can send the content via satellite or other media to a content distribution system serving a designated market area (DMA) within the U.S. Each such distribution system can then receive a national satellite feed carrying the video content and send it to televisions and / or set-top boxes within the DMA, allowing the content to be output and presented to the corresponding end users within that DMA. In practice, these content distribution systems and their transmission methods to content presentation devices can take various forms. For instance, a content distribution system can be associated with a cable television provider and can send video content to the content presentation devices of end users who are cable television subscribers via a hybrid fiber optic / coaxial cable connection.

[0007] Thus, in various scenarios, a content delivery system can send content to a content presentation device, which can receive and output the content to present it to the end user. In some cases, even if the content presentation device receives content from the content delivery system, it may be expected to perform content modification operations so that the content presentation device can output alternative content for presentation, rather than at least a portion of the received content.

[0008] For example, where a content presentation device receives a linear sequence of content segments including a given ad segment located somewhere within the sequence, it might be desirable for the content presentation device to replace the given ad segment with a different ad segment that is potentially more targeted to the end user (e.g., more targeted to the end user's interests, demographic characteristics, etc.). As another example, it might be desirable for the content presentation device to overlay content on a given ad segment in a manner that is also potentially more targeted to the end user. The described content modification system can facilitate the provision of these and other relevant features.

[0009] To facilitate the preparation and delivery of modified content to end users, components of the disclosed content modification system can perform fingerprinting on content accessed at different points within the system, thereby generating fingerprint data representing the accessed content. For example, a content distribution system can access and fingerprint upstream content (e.g., content that the system is currently sending or is about to send via a channel) and / or a content presentation device can access and fingerprint downstream content (e.g., content received by the device). The content modification system can compare the fingerprint data to detect matches, which helps identify where modifications should be made in a linear sequence of content segments, such as where to place the target content. Furthermore, in scenarios where the channel on which the content presentation device is receiving content is unknown, fingerprint matching can be used to identify the channel. Once the channel is identified, the content modification system can identify upcoming content modification opportunities within a linear sequence of content segments on the identified channel.

[0010] In practice, fingerprint matching and channel identification can be performed by a fingerprint matching server in the publicly disclosed content modification system. For example, the fingerprint matching server can receive reference fingerprint data from one or more content distribution systems, representing content being sent on an identified channel (e.g., a channel identified by metadata associated with the content being sent), and can also receive query fingerprint data representing content being received and / or output by a content presentation device on an unidentified channel. The fingerprint matching server can then compare the reference fingerprint data with the query fingerprint data to determine if a match exists. A match between the query fingerprint data and a specific reference fingerprint data enables the fingerprint matching server to identify the channel.

[0011] However, when the fingerprint matching server doesn't know which channel it's receiving and / or outputting the content represented by the query fingerprint data, it may need to compare large (e.g., hundreds) datasets of reference fingerprints associated with multiple different channels until a match is detected and the channel is identified. This can be a time-consuming process, and in some cases, may require significant storage and processing power from the fingerprint matching server to store and process the reference fingerprint datasets. Furthermore, when end users change channels, especially frequently, the fingerprint matching server often needs to repeat this process for the new channels, resulting in even greater processing power and latency when identifying new channels.

[0012] Usage and Terminology

[0013] In this disclosure, unless otherwise stated and / or explicitly specified in the particular context, the term "a" or "an" means at least one, and the term "the" means the at least one. Summary of the Invention

[0014] In one aspect, a method includes determining a content transmission latency between a content delivery system and a content presentation device. The method further includes selecting a reference fingerprint dataset corresponding to the determined content transmission latency from a plurality of reference fingerprint datasets, using at least the determined content transmission latency as a basis. The method also includes sending the selected reference fingerprint dataset corresponding to the determined content transmission latency to the content presentation device to facilitate the content presentation device's detection of a match between query fingerprint data representing content received by the content presentation device and at least a portion of the reference fingerprint data in the sent reference fingerprint dataset.

[0015] In another aspect, a non-transitory computer-readable storage medium stores program instructions that, when executed by a processor, cause a set of operations to be performed. The set of operations includes determining a content transmission delay between a content delivery system and a content presentation device. The set of operations also includes selecting a reference fingerprint dataset corresponding to the determined content transmission delay from a plurality of reference fingerprint datasets, using at least the determined content transmission delay as a basis. The set of operations further includes transmitting the selected reference fingerprint dataset corresponding to the determined content transmission delay to the content presentation device to facilitate the content presentation device's detection of a match between query fingerprint data representing content received by the content presentation device and at least a portion of the reference fingerprint data in the transmitted reference fingerprint dataset.

[0016] In another aspect, a computing system includes a processor and a non-transitory computer-readable storage medium thereon storing program instructions that, when executed by the processor, cause a set of operations to be performed. The set of operations includes determining a content transmission delay between a content delivery system and a content presentation device. The set of operations also includes selecting a reference fingerprint dataset corresponding to the determined content transmission delay from a plurality of reference fingerprint datasets, using at least the determined content transmission delay as a basis. The set of operations further includes transmitting the selected reference fingerprint dataset corresponding to the determined content transmission delay to the content presentation device to facilitate the content presentation device in detecting a match between query fingerprint data representing content received by the content presentation device and at least a portion of the reference fingerprint data in the transmitted reference fingerprint dataset. Attached Figure Description

[0017] Figure 1 It is a simplified block diagram of an example content modification system in which various described principles can be implemented.

[0018] Figure 2 It is a simplified block diagram of an example computing system in which various described principles can be implemented.

[0019] Figure 3 It is a graph of an example linear sequence of content and related concepts.

[0020] Figure 4A , 4B 4C, 4D, 4E, and 4F together form a table that shows the example time period and the corresponding operations performed by the system that can be modified based on the example content.

[0021] Figure 5 This is a flowchart of the example method.

[0022] Figure 6 This is a flowchart of the example method.

[0023] Figure 7 This is a flowchart of the example method. Detailed Implementation

[0024] I. Overview

[0025] This disclosure provides methods and systems for improving the fingerprint matching process by enabling the content presentation device to perform fingerprint matching and channel identification locally, and by reducing the number of reference fingerprint datasets stored and considered by the content presentation device during the matching process. Specifically, the disclosed methods and systems may involve a fingerprint matching server determining the content transmission delay between a content distribution system (e.g., a television broadcasting station) and the content presentation device, and using the determined content transmission delay as a basis to select a specific reference fingerprint dataset corresponding to the determined content transmission delay from a plurality of reference fingerprint datasets. The fingerprint matching server may then send the selected reference fingerprint dataset to the content presentation device, such that the content presentation device can then compare query fingerprint data with at least a portion of the reference fingerprint data from the sent reference fingerprint dataset to detect a match. In response to detecting a match, the content presentation device may perform actions such as identifying the channel on which the content presentation device is presenting or receiving content represented by the query fingerprint data, and / or may perform another action such as facilitating the content presentation device to perform a content modification operation.

[0026] By enabling the content presentation device to perform fingerprint matching and channel identification locally, the computational burden on the fingerprint matching server can be reduced. Furthermore, by using content transmission latency as the basis for selecting the corresponding reference fingerprint dataset, the number of reference fingerprint datasets sent by the fingerprint matching server to the content presentation device for storage and consideration during the matching process can be reduced. Reducing the number of reference fingerprint datasets in this way effectively reduces the number of channels the content presentation device considers when identifying the channels on which it receives and / or outputs content. This helps minimize the storage and processing power required for the content presentation device to store reference fingerprint datasets and identify channels. In contrast, without further selection of which fingerprint datasets should be sent to the content presentation device, the content presentation device might need to store and consider large (e.g., hundreds or thousands) reference fingerprint datasets, thus requiring significant storage and processing power.

[0027] In practice, the content transmission latency between a content delivery system and a specific content presentation device on a channel is typically roughly the same for most or all other channels broadcasting content to that content presentation device. Therefore, when an end user changes channels, the new channel usually has roughly the same or identical content transmission latency as the previous channel. Thus, a new channel can be one of several channels represented by a reference fingerprint dataset already selected based on the content transmission latency between the content delivery system and the content presentation device. Therefore, reducing the number of reference fingerprint datasets as described above also reduces the amount of processing power and time required for the content presentation device to identify channels when end users change channels and / or are changing channels at a high frequency, thereby enabling the content modification system to identify content modification opportunities on the current channel more quickly.

[0028] II. Architecture

[0029] Figure 1 This is a simplified block diagram of an example content modification system 100. The content modification system 100 may include various components, such as a content distribution system 102, a content presentation device 104, a fingerprint matching system server 106, a content management system 108, a data management system 110, and / or a supplementary content delivery system 112.

[0030] The content modification system 100 may also include one or more connection mechanisms that connect various components within the content modification system 100. For example, such as Figure 1 As shown, the content modification system 100 may include a connection mechanism represented by lines connecting the components of the content modification system 100.

[0031] In this disclosure, the term "connection mechanism" refers to a mechanism that connects and facilitates communication between two or more components, devices, systems, or other entities. A connection mechanism can be or include relatively simple mechanisms, such as cables or system buses, and / or relatively complex mechanisms, such as packet-based communication networks (e.g., the Internet). In some cases, a connection mechanism can be or include an intangible medium, such as where the connection is at least partially wireless. In this disclosure, a connection can be a direct connection or an indirect connection, the latter being a connection that passes through and / or traverses one or more entities, such as routers, switches, or other network devices. Similarly, in this disclosure, communication (e.g., sending or receiving data) can be direct communication or indirect communication.

[0032] The content modification system 100 and / or its components may take the form of a computing system, an example of which is described below.

[0033] It is worth noting that, in practice, the content modification system 100 may include many instances of at least some of the components mentioned above. For example, the content modification system 100 may include many content distribution systems and many content presentation devices.

[0034] B. Computing System

[0035] Figure 2 This is a simplified block diagram of an example computing system 200. The computing system 200 can be configured to perform and / or can perform one or more operations, such as those described in this disclosure. The computing system 200 may include various components, such as a processor 202, a data storage unit 204, a communication interface 206, and / or a user interface 208.

[0036] Processor 202 may be or include a general-purpose processor (e.g., a microprocessor) and / or a special-purpose processor (e.g., a digital signal processor). Processor 202 may execute program instructions included in the data storage unit 204 as described below.

[0037] Data storage unit 204 may be or include one or more volatile, non-volatile, removable, and / or non-removable storage components, such as magnetic, optical, and / or flash memory, and / or may be integrated wholly or partially with processor 202. Furthermore, data storage unit 204 may be or include a non-transitory computer-readable storage medium storing program instructions (e.g., compiled or uncompiled program logic and / or machine code) that, when executed by processor 202, cause computing system 200 and / or another computing system to perform one or more operations, such as those described in this disclosure. These program instructions may define a discrete software application and / or be part of a discrete software application.

[0038] In some cases, the computing system 200 can execute program instructions in response to receiving input, such as input received via the communication interface 206 and / or the user interface 208. The data storage unit 204 can also store other data, such as any data described in this disclosure.

[0039] Communication interface 206 allows computing system 200 to connect to and / or communicate with another entity according to one or more protocols. Therefore, computing system 200 can send data to one or more other entities and / or receive data from one or more other entities according to one or more protocols. In one example, communication interface 206 may be or include a wired interface, such as an Ethernet interface or a High-Definition Multimedia Interface (HDMI). In another example, communication interface 206 may be or include a wireless interface, such as a cellular or Wi-Fi interface.

[0040] User interface 208 allows interaction between computing system 200 and its user. Therefore, user interface 208 may be or include input components such as a keyboard, mouse, remote control, microphone, and / or touch panel. User interface 208 may also be or include output components such as a display device (which may be combined with a touch panel, for example) and / or a speaker.

[0041] The computing system 200 may also include one or more connection mechanisms connecting various components within the computing system 200. For example, such as Figure 2 As shown, the computing system 200 may include a connection mechanism represented by lines connecting the components of the computing system 200.

[0042] The computing system 200 may include one or more of the components described above and may be configured or arranged in various ways. For example, the computing system 200 may be configured as a server and / or client operating in one or more server-client type arrangements (or it may be a server cluster and / or client cluster).

[0043] As described above, the content modification system 100 and / or its components can take the form of a computing system such as computing system 200. In some cases, some entities or all of these entities can take the form of a more specific type of computing system. For example, in the case of content presentation device 104, it can take the form of a desktop computer, laptop computer, tablet computer, mobile phone, television, set-top box, television with integrated set-top box, media dongle, or television connected to a media dongle, among other possibilities.

[0044] III. Example Operation

[0045] The content modification system 100 and / or its components can be configured to perform and / or perform one or more operations. Examples of these operations and related features will now be described.

[0046] As described above, in practice, the content modification system 100 may include numerous instances of at least some of the components described. Similarly, in practice, at least some of the operations described may be performed multiple times (possibly based on routines and / or related to additional instances of the described components).

[0047] A. Operations related to the content delivery system sending content and the content presentation device receiving and outputting content.

[0048] For the context of this document, general operation and examples relating to the content distribution system 102 that sends content and the content presentation device 104 that receives and outputs content will now be described.

[0049] First, content distribution system 102 can send content (e.g., content received from a content provider) to one or more entities such as content presentation device 104. The content may be, for example, audio content and / or video content. In some examples, the content may take the form of a linear sequence of content segments (e.g., program segments and advertising segments) or portions thereof. For example, in the case of video content, a portion of the video content may be one frame or more frames.

[0050] Content distribution system 102 can send content on one or more channels (sometimes called stations or feeds). Therefore, content distribution system 102 can be associated with a single-channel content distributor or a multi-channel content distributor such as a multi-channel video program distributor (MVPD).

[0051] The content distribution system 102 and its means of transmitting content on a channel to the content presentation device 104 can take various forms. For example, the content distribution system 102 may be or include a cable television headend associated with a cable television provider and transmitting content on a channel to the content presentation device 104 via a hybrid fiber / coaxial cable connection. As another example, the content distribution system 102 may be or include a satellite television headend associated with a satellite television provider and transmitting content on a channel to the content presentation device 104 via satellite transmission. As yet another example, the content distribution system 102 may be or include a television broadcasting station associated with a television broadcasting provider and transmitting content on a channel to the content presentation device 104 via a terrestrial air interface. In these and other examples, the content distribution system 102 may transmit content in the form of an analog or digital broadcast stream representing the content.

[0052] Content presentation device 104 may receive content from one or more entities (e.g., content delivery system 102). In one example, content presentation device 104 may select (e.g., by tuning to) a channel from a plurality of available channels, possibly based on input received via a user interface, so that content presentation device 104 may receive content on the selected channel.

[0053] In some examples, content delivery system 102 can send content that content presentation device 104 can receive to content presentation device 104, so the sent content and the received content can be the same. However, in other examples, they can be different; for example, content delivery system 102 sends content to content presentation device 104, but content presentation device 104 does not receive content but receives different content from a different content delivery system.

[0054] Content presentation device 104 can also output content for presentation. As described above, content presentation device 104 can take various forms. In one example, if content presentation device 104 is a television (possibly with an integrated set-top box and / or media dongle), outputting content for presentation may involve the television outputting content via a user interface (e.g., a display device and / or speakers) so that it can be presented to an end user. As another example, if content presentation device 104 is a set-top box or media dongle, outputting content for presentation may involve the set-top box or media dongle outputting content via a communication interface (e.g., an HDMI interface) so that it can be received by the television and then output by the television for presentation to the end user.

[0055] Thus, in various scenarios, the content delivery system 102 can send content to the content presentation device 104, which can receive and output the content to present it to the end user. In some cases, even if the content presentation device 104 receives content from the content delivery system 102, it may be expected that the content presentation device 104 will perform a content modification operation so that the content presentation device 104 can output alternative content for presentation, rather than at least a portion of the received content.

[0056] For example, when content presentation device 104 receives a linear sequence of content segments including a given ad segment located somewhere within the sequence, it may be desirable for content presentation device 104 to replace the given ad segment with a different ad segment that is potentially more targeted to the end user (i.e., more targeted to the end user's interests, demographic characteristics, etc.). As another example, it may be desirable for content presentation device 104 to overlay content on a given ad segment in a manner that is also potentially more targeted to the end user. The described content modification system 100 can facilitate the provision of these and other related features.

[0057] As described above, in one example, the content can take the form of a linear sequence of content segments. Therefore, in one example, the content distribution system 102 can send a linear sequence of content segments. This is referred to herein as the "sending sequence". Similarly, the content presentation device 104 can receive a linear sequence of content segments. This is referred to herein as the "receiving sequence". Based on the discussion above, the sending sequence and the receiving sequence can be the same or different.

[0058] Figure 3 Some examples of these concepts are shown. In one example, the sending sequence is... Figure 3 The transmission sequence 302 is shown. As shown, transmission sequence 302 includes program segment A, followed by advertisement segment B, followed by advertisement segment C.

[0059] Similarly, in one example, the received sequence is Figure 3 The received sequence 304 is shown. In this example, the content distribution system 102 sends the transmitted sequence 302 to the content presentation device 104, which receives it as the received sequence 304. Therefore, the transmitted sequence 302 and the received sequence 304 are identical. Thus, as shown, the received sequence 304 also includes program segment A, followed by advertisement segment B, and then advertisement segment C.

[0060] exist Figure 3 In the diagram, the transmission time of transmission sequence 302 and the reception time of reception sequence 304 are shown through their relationship to timeline 350. It is worth noting that the transmission and reception times are offset from each other due to content transmission delays, which will be described in more detail below.

[0061] As described above, in some cases, even if the content presentation device 104 receives content from the content delivery system 102, it may be expected that the content presentation device 104 performs a content modification operation so that the content presentation device 104 can output alternative content for presentation, instead of at least a portion of the received content. For example, if the content presentation device 104 receives a received sequence but does not output that received sequence for presentation, the content presentation device 104 may instead output a modified version of the received sequence for presentation. This is referred to herein as a "modified sequence".

[0062] For example, in the case where the received sequence includes a given ad segment located somewhere within the received sequence, it may be desirable for the content presentation device 104 to replace the given ad segment with a different ad segment that may be more targeted to the end user (i.e., more targeted to the end user's interests, demographic characteristics, etc.), thereby producing a modified sequence that the content presentation device 104 can output for presentation.

[0063] To illustrate this, in one example, the modified sequence is: Figure 3 The first modification sequence 306 is shown. As shown, the first modification sequence 306 includes program segment A, followed by advertisement segment D (which replaces advertisement segment B), followed by advertisement segment C.

[0064] As another example, it may be expected that content presentation device 104 overlays content on a given ad segment in a manner that is also likely more targeted to the end user, thereby producing a modified sequence that content presentation device 104 can output for presentation.

[0065] To illustrate this, in another example, the modified sequence is Figure 3 The second modification sequence 308 is shown. As shown, the second modification sequence 308 includes program segment A, followed by advertisement segment B' (which is an advertisement segment B modified using overlay content), followed by advertisement segment C.

[0066] Figure 3 The remaining parts will be described later in this disclosure along with the introduction and description of the relevant concepts.

[0067] Considering the context provided above, Figure 4A , 4B 4C, 4D, 4E, and 4F together form a table that shows example time periods and the corresponding operations that can be performed by the content modification system 100. These and other related operations will now be described.

[0068] B. Operations related to the content distribution system sending the first content on a channel.

[0069] During time period Tl, content delivery system 102 may send content to content presentation device 104 on a channel. This content is referred to herein as "first content". In one example, the first content is... Figure 3 The first content shown is 310.

[0070] During time period T2, content delivery system 102 may generate fingerprint data representing first content. This fingerprint data is referred to herein as "first fingerprint data". Content delivery system 102 may use any content fingerprinting process now known or developed hereafter to generate the first fingerprint data. Example fingerprint generation techniques are described in U.S. Patent No. 9,495,451, published November 15, 2016, the entire contents of which are incorporated herein by reference. Content delivery system 102 may generate the first fingerprint data at a given rate, for example, at a rate of one fingerprint per frame of first content. The first fingerprint data may be or include some or all of these generated fingerprints.

[0071] Content delivery system 102 can access first content at different points within content delivery system 102. As an example, content delivery system 102 can access the first content after it has been output by a distribution amplifier within content delivery system 102.

[0072] Also during time period T2, content delivery system 102 may generate metadata associated with the first content and / or the first fingerprint data. This metadata is referred to herein as "first metadata". In one example, the first metadata may be or include a transmission timestamp, which indicates the point in time when content delivery system 102 transmits the first content. Content delivery system 102 may determine the transmission timestamp in various ways, such as based on a clock synchronized with a reference clock.

[0073] As another example, the first metadata may be or include a channel identifier that identifies the channel on which the content delivery system 102 is sending the first content. The content delivery system 102 may determine the channel identifier in various ways, such as based on mapping data that maps the physical inputs and / or outputs within the content delivery system 102 to corresponding channel identifiers. In one example, where the content delivery system 102 sends content A on channel A, content B on channel B, and content C on channel C, the mapping data may specify which of the three different outputs (possibly on three different distribution amplifiers) maps to which channel identifier, allowing the content delivery system 102 to determine the appropriate channel identifier for the content of a given channel.

[0074] As another example, the first metadata may be or include SCTE-104 data, watermarks, or similar types of metadata, any of which may encode other metadata, such as program identifiers, advertising identifiers (e.g., Industry Standard Code Identifier (ISCI) keys), program genres, or other types of text or digital metadata.

[0075] The content distribution system 102 can associate the first fingerprint data with the first metadata in various ways. For example, if the first fingerprint data includes multiple fingerprints and each fingerprint represents a corresponding frame of the first content, the content distribution system 102 can associate each fingerprint with a corresponding sending timestamp and / or other corresponding first metadata.

[0076] During time period T3, content distribution system 102 may send the first fingerprint data and first metadata to fingerprint matching server 106. Content distribution system 102 may send the first fingerprint data and first metadata at given intervals. For example, content distribution system 102 may send the first fingerprint data and first metadata generated within the most recent two-second time period every two seconds.

[0077] C. Operations related to the content presentation device receiving second content

[0078] During time period T4, content presentation device 104 can receive content from content distribution system 102. This content is referred to herein as "second content". In one example, the second content is... Figure 3 The second content 312 is shown.

[0079] During time period T5, content rendering device 104 may generate fingerprint data representing the second content. This fingerprint data is referred to herein as "second fingerprint data". Content rendering device 104 may use any content fingerprinting process now known or developed hereafter to generate the second fingerprint data. Content rendering device 104 may generate the second fingerprint data at various rates, such as one fingerprint per frame of the second content. The second fingerprint data may be or include some or all of these generated fingerprints.

[0080] Content presentation device 104 can access second content at different points within content presentation device 104. As an example, content presentation device 104 can access the second content while it is being received via its input buffer (e.g., an HDMI buffer). In another configuration, content presentation device 104 can access the second content while it is being received by its display buffer. Therefore, in this configuration, the second content can be not only content received by content presentation device 104, but also content output for presentation.

[0081] Also during time period T5, content presentation device 104 may generate metadata associated with the second content and / or second fingerprint data. This metadata is referred to herein as "second metadata". As an example, the second metadata may be or include a reception timestamp, which indicates the point in time when content presentation device 104 received the second content. Content presentation device 104 may determine the reception timestamp in various ways, such as based on a clock synchronized with a reference clock. As described above, content presentation device 104 may access the second content at different points within content presentation device 104. In one example, to determine the reception timestamp, the point where the second content is accessed may be considered the "reception" point.

[0082] In practice, when the first metadata is likely to be or include the channel identifier, the second metadata is likely to be neither or include the channel identifier.

[0083] The content presentation device 104 can associate the second fingerprint data with the second metadata in various ways. For example, when the second fingerprint data includes multiple fingerprints and each fingerprint represents a corresponding frame of the second content, the content presentation device 104 can associate each second fingerprint with a corresponding reception timestamp and / or other corresponding metadata.

[0084] During time period T6, content presentation device 104 may send second fingerprint data and second metadata to fingerprint matching server 106. Content presentation device 104 may send the second fingerprint data and second metadata at given intervals. For example, content presentation device 104 may send the second fingerprint data and second metadata generated within the most recent two-second time period every two seconds.

[0085] D. Operations related to the channel on which the content presentation device is receiving second content.

[0086] During time period T7, fingerprint matching server 106 can receive first fingerprint data and first metadata from content distribution system 102. As described above, the first fingerprint data represents the first content sent by content distribution system 102 on a channel. As described above, the first metadata can (and is intended to be used in this described example) identify the channel. Thus, the first content can be considered as content sent on the identified channel.

[0087] During time period T8, fingerprint matching server 106 can receive second fingerprint data and second metadata from content presentation device 104. As described above, the second fingerprint data represents the second content received by content presentation device 104. However, as described above, the associated metadata may (and in this example, for the purpose of) not identify the channel. Thus, the second content could be considered as content received on an unidentified channel.

[0088] During time period T9, fingerprint matching server 106 can compare first fingerprint data and second fingerprint data to determine whether a match exists. In this disclosure, this type of matching attempt, namely a matching attempt between (i) reference fingerprint data representing content being sent on an identified channel and (ii) query fingerprint data representing content being received on an unidentified channel, is referred to herein as a "cold matching attempt".

[0089] During time period T10, fingerprint matching server 106 can detect a match between first fingerprint data and second fingerprint data based on comparison. Fingerprint matching server 106 can use any fingerprint comparison and matching technique now known or developed hereafter to compare and / or detect matches between fingerprint data. Example fingerprint comparison and matching techniques are described in U.S. Patent No. 9,146,990, published September 29, 2015, the entire contents of which are incorporated herein by reference.

[0090] To effectively compare the first fingerprint data with the second fingerprint data, the fingerprint matching server 106 may need to consider content transmission delay. In this context, if the content distribution system 102 sends a given content frame on a given channel at time A, the content presentation device 104 may not receive the frame until time B, which is later than time A (e.g., 10 seconds later), for various reasons. This type of delay is referred to herein as "content transmission delay".

[0091] In one example, such as Figure 3 As shown, time point A, time point B, and content transmission delay can be time point A314, time point B316, and content transmission delay318, respectively. It should be noted that... Figure 3 For illustrative purposes only and not necessarily scaled proportionally, at least in terms of time. In reality, the actual content delivery latency may differ from that shown.

[0092] To help the fingerprint matching server 106 effectively compare the first fingerprint data with the second fingerprint data, the fingerprint matching server 106 may need to account for this content transmission delay. In one example, the fingerprint matching server 106 can do this by comparing the first fingerprint data it receives at the receiving time with the second fingerprint data it receives within a time period defined by a start time and an end time. The start time can be the receiving time plus an offset representing the expected content transmission delay (e.g., ten seconds), minus a tolerance for the time period (e.g., two seconds). The end time can be the receiving time plus the offset (e.g., ten seconds), plus a tolerance for the time period (e.g., two seconds). Therefore, in an example where the expected content transmission delay is 10 seconds, the fingerprint matching server 106 can compare the first fingerprint data it receives at the receiving time with the second fingerprint data it receives within a time period from (i) the receiving time plus eight seconds to (ii) the receiving time plus twelve seconds.

[0093] In some cases, the fingerprint matching server 106 can determine the content transmission delay, which, as described above, can be used to select an appropriate offset for determining the start and end time points. The fingerprint matching server 106 can determine the content transmission delay in various ways. For example, after the fingerprint matching server 106 detects a match based on a cold match attempt, it can, for instance, determine the content transmission delay as the difference between the corresponding send timestamp (of the first metadata) and the corresponding receive timestamp (of the second metadata). It is worth noting that the content transmission delay may vary by channel.

[0094] During the time period Tll, based on the detected match, the fingerprint matching server 106 can identify the channel on which the content presentation device 104 is receiving second content. In one example, the fingerprint matching server 106 can identify the channel based on channel identifier metadata associated with the first fingerprint data used to detect the match.

[0095] It is worth noting that, in practice, since there may be multiple potential channels on which the content presentation device 104 receives the second content, the fingerprint matching server 106 may compare the second fingerprint data with multiple instances of the first fingerprint data (each representing a different corresponding instance of the first content on a different corresponding channel) to determine which of the multiple instances matches the second fingerprint data.

[0096] Furthermore, in some cases, the fingerprint matching server 106 can detect a match between the second fingerprint data and each of multiple instances of the first fingerprint data (each representing a different corresponding instance of the first content on a different corresponding channel). This is referred to herein as a "multi-match scenario" and can occur for various reasons. For example, this scenario can occur when the content delivery system 102 sends the same or similar content simultaneously or approximately simultaneously on more than one channel. In this scenario, the fingerprint matching server 106 can perform additional operations to identify from the multiple channels associated with the multi-match scenario which specific channel the content presentation device 104 is receiving the second content on. The fingerprint matching server 106 can do this using any channel multi-match disambiguation technique now known or developed hereafter. An example channel multi-match disambiguation technique is described in U.S. Patent No. 9,992,533, published June 5, 2018, the entire contents of which are incorporated herein by reference.

[0097] E. Operations related to determining historical content consumption data

[0098] During time period T12, fingerprint matching server 106 can generate metadata associated with the identified channel. For example, the metadata may be or include a channel identification timestamp. Fingerprint matching server 106 can determine the channel identification timestamp in various ways, such as based on a clock synchronized with a reference clock. In another example, the metadata may be or include a device identifier identifying the content presentation device 104 that is receiving content on the identified channel. Fingerprint matching server 106 can determine the device identifier in various ways, such as by receiving the device identifier from the content presentation device 104. In another example, fingerprint matching server 106 may receive data (e.g., device registration data) from the content presentation device 104 and may use mapping data to map the received data to determine the device identifier.

[0099] During time period T13, fingerprint matching server 106 can send an indication of the identified channel and associated metadata to data management system 110.

[0100] During time period T14, the data management system 110 can receive an indication of the identified channel and associated metadata from the fingerprint matching server 106.

[0101] The data management system 110 can use the received indications of the identified channel and associated metadata, and perhaps other data, to determine when the content presentation device 104 received content on the identified channel, what specific content the content presentation device 104 has received, and so on. This type of data is referred to herein as "historical content consumption data".

[0102] In this way, during time period T15, the data management system 110 can determine the historical content consumption data associated with the content presentation device 104.

[0103] F. Operations related to the content distribution system sending third-party content

[0104] As described above, the fingerprint matching server 106 can identify the channel on which the content presentation device 104 is receiving second content.

[0105] During time period T16, content delivery system 102 may send content to content presentation device 104 on the identified channel. This content is referred to herein as "third content." In one example, the third content is... Figure 3 The third content 320 is shown. In practice, the content distribution system 102 may send the third content shortly after the first content (e.g., immediately after it, or a few seconds or minutes later).

[0106] During time period T17, content distribution system 102 may generate fingerprint data representing third content. This fingerprint data is referred to herein as "third fingerprint data".

[0107] Also during time period T17, content distribution system 102 can generate metadata associated with third content and / or third fingerprint data. This metadata is referred to herein as "third metadata". Content distribution system 102 can also associate third fingerprint data with third metadata.

[0108] During time period T18, content distribution system 102 can send third fingerprint data and third metadata to fingerprint matching server 106.

[0109] The content distribution system 102 can send third content, generate third fingerprint data, generate third metadata, associate third fingerprint data with third metadata, and send third fingerprint data and third metadata in various ways, for example in the same or similar ways as those described above for sending first content, generating first fingerprint data, generating first metadata, associating first fingerprint data with first metadata, and sending first fingerprint data and first metadata.

[0110] G. Operations related to the content management system receiving modifiable content segments.

[0111] During time period T19, the content management system 108 can receive content in the form of candidate content segments that have been identified as needing modification. This content is referred to herein as a "modifiable content segment" or "fourth content." In one example, the modifiable content segment is... Figure 3 The editable content section is shown.

[0112] Modifiable content segments can take various forms. For example, a modifiable content segment can be an advertising segment (e.g., a commercial) or a program segment. Thus, in one example, a modifiable content segment could be an advertising segment that has been identified as a candidate for modification, possibly by replacing it with a different advertising segment, and / or by overlaying content on it.

[0113] In one example, a user who may be associated with content distribution system 102 may facilitate the uploading of modifiable content segments to content management system 108, so that content management system 108 can receive the modifiable content segments in this manner.

[0114] During time period T20, content management system 108 may generate fingerprint data representing a modifiable content segment. This fingerprint data is referred to herein as "fourth fingerprint data". Content management system 108 may use any fingerprint generation technology now known or developed hereafter to generate the fourth fingerprint data. Content management system 108 may generate the fourth fingerprint data at a given rate, for example, at a rate of one fingerprint per frame of fourth content. The fourth fingerprint data may be or include some or all of these generated fingerprints.

[0115] Also during time period T20, the content management system 108 can generate metadata associated with the modifiable content segment and / or fourth fingerprint data. This metadata is referred to herein as "fourth metadata." As an example, the fourth metadata may be or include the duration of the modifiable content segment. The content management system 108 can determine this duration in various ways, such as based on the fingerprint generation process. For example, in a case where the content management system 108 generating the fourth fingerprint data involves generating one fingerprint per frame, where the modifiable content segment has a frame rate of 30 frames per second, and where the fingerprint recognition process results in the generation of 300 fingerprints, then the content management system 108 can infer that the modifiable content segment has a duration of ten seconds. The metadata may also be or include other information about the modifiable content segment, such as the content segment identifier, title, and / or specifications regarding the permitted ways in which the modifiable content segment can be modified.

[0116] During time period T21, the content management system 108 can send the fourth fingerprint data and the fourth metadata to the fingerprint matching server 106.

[0117] In practice, the content management system 108 may receive many modifiable content segments. In this case, the content management system 108 may, as appropriate, perform one or more of the above-described operations on each of the many received modifiable content segments. Thus, the content management system 108 may send many instances of fourth fingerprint data to the fingerprint matching server 106, each instance corresponding to a different corresponding modifiable content segment.

[0118] H. Operations related to the fingerprint matching server identifying upcoming content modification opportunities on the identified channels.

[0119] During time period T22, fingerprint matching server 106 can receive third fingerprint data and third metadata from content distribution system 102. As described above, the third fingerprint data represents third content sent by content distribution system 102 on the identified channel.

[0120] During time period T23, fingerprint matching server 106 can receive fourth fingerprint data and fourth metadata from content management system 108. As described above, the fourth fingerprint data represents a modifiable content segment.

[0121] During time period T24, fingerprint matching server 106 can compare at least a portion of the third fingerprint data with at least a portion of the fourth fingerprint data to determine whether a match exists.

[0122] During time period T25, based on this comparison, fingerprint matching server 106 can detect a match between at least a portion of the third fingerprint data and at least a portion of the fourth fingerprint data. Fingerprint matching server 106 can use any fingerprint comparison and matching process now known or developed in the future to compare and / or detect matches between fingerprint data.

[0123] During time period T26, based on the detected matches, fingerprint matching server 106 can determine that at least a portion of the modifiable content segment is included within the third content, and thus can identify upcoming content modification opportunities on the identified channel. For example, as Figure 3 As shown, the fingerprint matching server 106 can determine that at least the beginning of the modifiable content segment is included within the third content 320, and thus can identify an upcoming opportunity to modify the content.

[0124] As described above, the fingerprint matching server 106 can generate third metadata, which may be the same as or similar to the first metadata. Therefore, the third metadata may be, or include, for example, a transmission timestamp and / or a channel identifier. However, the third metadata may also be, or include, the location of at least a portion of a modifiable content segment within the third content. In one example, the metadata may use a start frame marker and an end frame marker to specify this location, each marker corresponding to a corresponding frame of the third content. The fingerprint matching server 106 can determine the start frame marker and end frame marker based on the match.

[0125] It is worth noting that, in practice, since there may be multiple potential modifiable content segments, parts of which may be included in the third content, the fingerprint matching server 106 may compare at least a portion of the third fingerprint data with at least a portion of multiple instances of the fourth fingerprint data (each instance representing a different corresponding instance of the modifiable content segment) to determine which of the multiple instances of the fourth fingerprint data has a part that matches at least a portion of the third fingerprint data.

[0126] I. Operations related to preparing the content presentation device to perform content modification operations associated with identified upcoming content modification opportunities.

[0127] During time period T27, based on the detected match, fingerprint matching server 106 can send third fingerprint data and third metadata to content presentation device 104 to facilitate preparation of content presentation device 104 for performing content modification operations related to the identified upcoming content modification opportunities.

[0128] During time period T28, content presentation device 104 can receive third fingerprint data and third metadata from fingerprint matching server 106.

[0129] During time period T29, content presentation device 104 can receive content on the identified channel. This content is referred to herein as "the fifth content." In one example, the fifth content is... Figure 3 The fifth item shown is 324.

[0130] For various reasons (e.g., because the transmission latency associated with sending fingerprint data and metadata is shorter than the content transmission latency), content presentation device 104 can receive third fingerprint data and third metadata from fingerprint matching server 106 before receiving fifth content from content distribution system 102. In this way, content presentation device 104 can receive fingerprint data indicating content that content presentation device 104 expects to receive shortly thereafter, and unless an interruption event occurs (e.g., a channel change event), the content presentation device should actually receive that content shortly thereafter.

[0131] In practice, similar to how content distribution system 102 might send third content shortly after sending the first content (e.g., immediately after, or a few seconds or minutes later), content presentation device 104 might receive fifth content shortly after receiving the second content (e.g., immediately after, or a few seconds or minutes later).

[0132] During time period T30, content presentation device 104 can output at least a portion of the fifth content for presentation. For example, see reference. Figure 3 The content presentation device can output a portion of the fifth content 324, which is the end of program segment A, for presentation.

[0133] As described above, in some cases, even if the content presentation device 104 receives content from the content distribution system 102, it may be expected that the content presentation device 104 performs a content modification operation so that the content presentation device 104 can output alternative content for presentation, rather than at least a portion of the received content.

[0134] Thus, even if the content presentation device 104 receives the fifth content and outputs at least a portion of the fifth content for presentation, it may be expected that the content presentation device 104 performs a content modification operation so that the content presentation device 104 can also output alternative content for presentation, instead of at least another portion of the fifth content (e.g., the remainder). For example, see reference... Figure 3 It may be desirable for content presentation device 104 to replace at least a portion of ad segment B with at least a portion of a different ad segment that may be more targeted to the end user. As another example, it may be desirable for content presentation device 104 to overlay at least a portion of ad segment B with content that enhances the coverage of at least a portion of ad segment B in a manner that is also likely more targeted to the end user.

[0135] During time period T31, content presentation device 104 may generate fingerprint data representing the fifth content. This fingerprint data is referred to herein as "fifth fingerprint data". Content distribution system 102 may use any content fingerprinting process now known or developed hereafter to generate the fifth fingerprint data. Content presentation device 104 may generate the fifth fingerprint data at various rates, such as one fingerprint per frame of the fifth content. The fifth fingerprint data may be or include some or all of these generated fingerprints.

[0136] Also during time period T31, content presentation device 104 may generate metadata associated with the fifth content and / or the fifth fingerprint data. This metadata is referred to herein as "fifth metadata".

[0137] The content presentation device 104 can receive fifth content, generate fifth fingerprint data, generate fifth metadata, and associate the fifth fingerprint data with the fifth metadata in various ways, such as in the same or similar ways as described above in combination with receiving second content, generating second fingerprint data, generating second metadata, and associating the second fingerprint data with the second metadata.

[0138] As described above, the content presentation device 104 can receive third fingerprint data from the fingerprint matching server 106 and can generate fifth fingerprint data.

[0139] During time period T32, content presentation device 104 can compare third fingerprint data with fifth fingerprint data to determine if a match exists.

[0140] During time period T33, based on this comparison, content presentation device 104 can detect a match between the third fingerprint data and the fifth fingerprint data. In this disclosure, this type of matching attempt, i.e., a matching attempt between (i) reference fingerprint data representing content sent by content distribution system 102 on an identified channel (at least based on the most recent channel identification analysis) and (ii) query fingerprint data representing content being received by content presentation device 104 on the same identified channel, is referred to herein as a "hot match attempt". Fingerprint matching server 106 can use any content fingerprint comparison and matching process now known or developed hereafter to compare and / or detect matches between fingerprint data.

[0141] During time period T34, based on the detected matches, content presentation device 104 can determine the start time of the identified upcoming modification opportunity. This is referred to herein as the "modification start time". In one example, the modification start time is as follows: Figure 3 The modification start time shown is 326.

[0142] In one example, the content presentation device 104 can determine the modification start time by starting with a transmission timestamp associated with the start frame marker (as described above, which may be or is included in third metadata) and adding the content transmission delay to that transmission timestamp.

[0143] As another example, firstly, the content presentation device 104 can determine the modification start time by establishing a synchronization lock between third content, third fingerprint data, and / or third metadata on one side and fifth content, fifth fingerprint data, and / or fifth metadata on the other side. The content presentation device 104 can establish the synchronization lock using any synchronization lock technique now known or developed hereafter. An example synchronization lock technique is described in U.S. Patent No. 10,506,275, published December 10, 2019, the entire contents of which are incorporated herein by reference. Another example synchronization lock technique is described in U.S. Patent Application Publication No. 2020 / 0029108, published January 23, 2020, the entire contents of which are incorporated herein by reference.

[0144] Then, the content presentation device 104 can determine the modification start time by determining (i) the current reception timestamp associated with the first part of the fifth content that the content presentation device 104 is currently receiving and (ii) the expected reception timestamp associated with the second part of the fifth content that is the start of the modifiable content segment based on synchronization lock, and then determining the modification start time by adding the determined time period to the current reception timestamp.

[0145] Also during time period T34, based on the detected matches, content presentation device 104 can determine the end time of the identified upcoming modification opportunity. This is referred to herein as the "modification end time". In one example, the modification end time is as follows: Figure 3 The modification ended at time 328.

[0146] In one example, the content rendering device 104 can determine the modification end time by the duration of the content segment that is modifiable at the modification start time (as described above, it may be or be included in the fourth metadata).

[0147] It is worth noting that if the content presentation device 104 performs a hot match attempt and no match is detected, the content presentation device 104 can determine that it no longer receives content on the most recently identified channel. In response, the content presentation device 104 can repeat one or more of the above operations, allowing the fingerprint matching server 106 to perform additional cold match attempts to try to identify the channel again.

[0148] During time period T35, content presentation device 104 may send a request to content management system 108 for content related to performing a content modification operation. This content is referred to herein as "supplementary content". In one example, content presentation device 104 may send the request before the modification start time (e.g., ten seconds prior). In some cases, the request may include selection criteria for the supplementary content; for example, content presentation device 104 may determine the selection criteria based on third-party metadata received by content presentation device 104 from fingerprint matching server 106.

[0149] For example, selection criteria can specify the type of content requested (e.g., replacement content segment or overridden content), duration (e.g., 15 seconds, 30 seconds or 60 seconds), aspect ratio (e.g., 4:3 or 16:9) and / or resolution (e.g., 720p or 1080p).

[0150] During time period T36, content management system 108 may receive the request and use it as the basis for selecting supplementary content from a plurality of available supplementary content items. In some cases, content management system 108 may receive and consider various data to help inform which supplementary content to select. For example, content management system 108 may receive historical content consumption data from content presentation device 104 from data management system 110, and / or content management system 108 may receive demographic data from a demographic data provider. Then, content management system 108 may use at least the received historical content consumption data and / or the received demographic data as the basis for selecting supplementary content.

[0151] Content management system 108 can send selected supplementary content to content presentation device 104. In one example, content management system 108 can achieve this by communicating with supplementary content delivery system 112, which can host the supplementary content. Supplementary content delivery system 112 can take various forms and can include various components such as content delivery network (CDN).

[0152] During time period T37, content management system 108 may send a request to supplementary content delivery system 112 for a link (e.g., a Uniform Resource Identifier (URI) or Uniform Resource Locator (URL)) to the hosted supplementary content.

[0153] During time period T38, the supplementary content delivery system 112 can receive and respond to requests for links by sending the requested links to the content management system 108.

[0154] During time period T39, the content management system 108 can then send the link to the content presentation device 104.

[0155] During time period T40, content presentation device 104 can receive a link that it can use to retrieve supplementary content from supplementary content delivery system 112, enabling content presentation device 104 to use the retrieved supplementary content related to performing content modification operations. In one example, content presentation device 104 can retrieve supplementary content and store it in its data storage unit.

[0156] Thus, in some examples, content presentation device 104 may receive modifiable content segments from one source (e.g., content distribution system 102) and supplementary content from another source (e.g., supplementary content delivery system 112). These segments may be sent to and received by content presentation device 104 in different ways. For example, content distribution system 102 may send modifiable content segments as broadcast streams, which content presentation device 104 may receive, while supplementary content delivery system 112 may send supplementary content as OTT (over-the-top) transmission, which content presentation device 104 may receive. In this context, in one example, content distribution system 102 may receive modifiable content segments via a communication interface (e.g., an HDMI interface), and content presentation device 104 may receive supplementary content via different communication interfaces (e.g., Ethernet or Wi-Fi interfaces).

[0157] J. Operations related to content modification performed by the content presentation device.

[0158] During time period T41, content presentation device 104 can perform content modification operations. Content presentation device 104 can do this in various ways, depending on the type of content modification operation to be performed.

[0159] In one example, a content modification operation performed by the content rendering device 104 may involve modifying a modifiable content segment by replacing it with supplementary content. This is referred to herein as a "content replacement operation." For example, in this case, the content rendering device 104 may receive a linear sequence of content segments including the modifiable content segment and associated metadata, and as described above, may also receive supplementary content segments. The content rendering device 104 may output and render the sequence of content segments up to the modification start time (which corresponds to the start of the modifiable content segment), at which point the content rendering device 104 may switch to output and render the supplementary content. Then, at the modification end time (which corresponds to the end of the modifiable content segment), the content rendering device 104 may switch back to output and render subsequent content in the linear sequence of content segments (or possibly other content, such as additional supplementary content replacing another modifiable content segment).

[0160] In one example, the operation of the content rendering device 104 switching from a sequence of output content segments to output supplementary content may involve using various buffers of the content rendering device 104. For example, this may involve the content rendering device 104 switching from using first data in a first input buffer that is receiving a sequence of content segments to using second data in a second input buffer that is receiving supplementary content, in order to input data into a display buffer.

[0161] Therefore, according to Figure 3 As shown in one example, by performing a content replacement operation, content rendering device 104 can replace ad segment B with ad segment D. As a result, the content rendering device can instead output a first modified sequence 306 for rendering, instead of outputting a received sequence 304 for rendering.

[0162] In another example, the content rendering device 104 performing a content modification operation may involve the content rendering device 104 modifying the modifiable content segment by overlaying overlay content onto the modifiable content segment (referred to herein as a "content overlay operation"). For example, in this case, the content rendering device 104 may similarly receive a linear sequence of content segments including the modifiable content segment and associated metadata, and the content rendering device 104 may also receive supplementary content as described above.

[0163] Then, the content rendering device 104 can output and render the modifiable content segment as normally would, except that, starting at the modification start time, the content rendering device 104 can begin to overlay supplementary content onto the modifiable content segment. The content rendering device 104 can continue to overlay the supplementary content until the modification end time. In this way, the content rendering device 104 can overlay the supplementary content for at least some time portions of the modifiable content segment.

[0164] In one example, the operation of the content rendering device 104 overlaying supplementary content on a modifiable content segment may involve using various buffers of the content rendering device 104. For example, for the purpose of inputting data into the display buffer, this may involve the content rendering device 104 using a portion of first data in a first input buffer that is receiving a sequence of content segments, along with second data in a second input buffer that is receiving supplementary content. In this way, the content rendering device can combine the relevant portions of the modifiable content segment (i.e., all portions except those representing the area to be overlaid with supplementary content) with the supplementary content to be used as overlay to create the desired modifiable content segment and the supplementary content overlaid thereon.

[0165] Therefore, according to such Figure 3 As shown in one example, by performing a content overlay operation, content rendering device 104 can overlay supplementary content onto ad segment B, thereby modifying it into ad segment B'. As a result, the content rendering device can instead output a second modified sequence 308 for rendering, instead of outputting an accepted sequence 304 for rendering.

[0166] K. Tracking and reporting operational data

[0167] To facilitate the execution of various operations (such as content presentation device 104 performing content modification operations), and to help allow for the tracking and reporting of these operations, content modification system 100 and / or its components may track and report various operation-related data at different times and in different ways.

[0168] As illustrative examples, in response to certain operations being performed, such as those described herein, fingerprint matching server 106, content presentation device 104, and / or another entity may generate, store, and / or send (i) messages indicating that a modifiable content segment has been identified, (ii) messages indicating that a channel has been identified / confirmed (possibly based on a match detected as a result of a cold match attempt or a hot match attempt), (iii) messages indicating that an upcoming content modification opportunity has been identified on the identified channel, (iv) messages indicating that supplementary content has been requested, (v) messages indicating that supplementary content has been received, (vi) messages indicating that a content modification operation has begun, (vii) messages indicating that a content modification operation has ended, and / or (viii) messages indicating that a planned content modification operation has been aborted and / or not executed for any given reason. In some cases, these messages may include additional metadata related to these operations. For example, metadata may specify relevant time information, device identifiers, channel identifiers, content segment identifiers, etc.

[0169] L. Watermarking-based technology

[0170] Although this disclosure has described a content modification system 100 that uses fingerprint-based technology to perform various operations and provide various features, in some examples, the content modification system 100 may use watermark-based technology in addition to fingerprint-based technology, or in addition to fingerprint-based technology, the content modification system 100 may also use watermark-based technology to perform these and other operations and provide these and other features.

[0171] For example, as an alternative to the fingerprint-based technology described above, where the fingerprint matching server 106 identifies the channel on which the content presentation device 104 is receiving the second content, the content distribution system 102 or another entity may insert a channel identifier in the form of a watermark into the second content, so that the fingerprint matching server 106, the content presentation device 104 or the other entity can extract the channel identifier and use it to identify the channel on which the content presentation device 104 is receiving the second content.

[0172] In this context, the content modification system 100 can employ any watermarking technology now known or developed in the future.

[0173] M. Operations related to the selection of a reference fingerprint dataset based on content delivery delay.

[0174] As described above, when the channel for which the content is received by the content presentation device is not identified, the fingerprint matching server 106 may need to perform numerous cold match attempts between (i) query fingerprint data representing such content and (ii) a large number of reference fingerprint datasets to determine whether the query fingerprint data matches at least a portion of any of the reference fingerprint datasets. To achieve this, the fingerprint matching server 106 may need to access (e.g., receive from the content delivery system 102) many different reference fingerprint datasets, and in some cases, store many different reference fingerprint datasets in memory, each representing one or more different instances of content being sent by the content delivery system on one or more corresponding channels. These numerous cold match attempts can be time-consuming and, in some cases, computationally expensive for the fingerprint matching server 106 in terms of memory and / or processing power. The time and computational costs can be even greater when end users frequently change channels.

[0175] Therefore, the disclosed method and system advantageously reduce the burden on the fingerprint matching server 106 by enabling the content presentation device 104 to perform fingerprint matching and channel identification. Additionally, according to the disclosed method and system, the fingerprint matching server 106 can be configured to perform and / or can perform one or more actions to reduce the amount of reference fingerprint dataset stored and used by the content presentation device 104 in performing cold matching attempts to identify channels. As a result, the number of cold matching attempts can be reduced, and the amount of time, storage, and processing power required for the content presentation device 104 to perform the fingerprint matching process can be reduced. Furthermore, because the content presentation device 104 identifies channels more quickly, it can send indications of the identified channels to the fingerprint matching server 106 more quickly, thereby accelerating the remaining portion of the process where the content modification system 100 identifies content modification opportunities on the identified channels and the content presentation device 104 takes action based on these identified content modification opportunities.

[0176] The fingerprint matching server 106 can determine the content transmission delay between the content distribution system 102 and the content presentation device 104. Determining the content transmission delay may involve determining the delay as the difference between the timestamp of the content sent by the content distribution system 102 and the timestamp of the same content received by the content presentation device 104. For example, the sent content may be first content, and the received content may be second content, where the second content is identical to the first content. Therefore, the timestamp may be the timestamp included in the first metadata discussed above, and the timestamp may be the timestamp included in the second metadata discussed above. In alternative embodiments, such as those where other timestamps are determined as the points of sending the first content and / or receiving the second content, other timestamps may be determined and used to determine the content transmission delay.

[0177] The action of determining content delivery delay can also take other forms. For example, determining content delivery delay may involve identifying one or more specific types of information, such as information associated with content presentation device 104, and mapping the identified information to content delivery delays. To achieve this, fingerprint matching server 106 may store in memory, and / or receive from other devices within or outside content modification system 100, mapping data (e.g., a table) that maps one or more different types of information associated with content presentation device 104 to corresponding content delivery delays. This mapping data is referred to herein as "first mapping data".

[0178] For example, fingerprint matching server 106 can identify the device identifier of content presentation device 104 and map the identified device identifier to content transmission latency using first mapping data. As described above, fingerprint matching server 106 can determine the device identifier in various ways, such as by receiving from content presentation device 104, and / or by receiving other data from content presentation device 104 and mapping the received data using other mapping data to determine the device identifier.

[0179] As another example, fingerprint matching server 106 can identify the IP address of content presentation device 104 and map the identified IP address to content transmission latency using first mapping data. Fingerprint matching server 106 can determine the IP address in various ways, such as by receiving data from content presentation device 104 or other devices within or outside content modification system 100, and / or by receiving additional data from content presentation device 104 or other devices and using additional mapping data to map the received data to determine the IP address.

[0180] As another example, fingerprint matching server 106 can identify content that is currently being presented, has been presented previously, and / or is planned to be presented on content presentation device 104, and map the identified content to a content transmission delay using first mapping data. Fingerprint matching server 106 can determine the content identifier in various ways, such as by receiving it from content presentation device 104 or other devices within or outside content modification system 100, and / or by receiving additional data from content presentation device 104 or other devices and using additional mapping data to map the received data to determine the content identifier. In some examples, the content identifier may be metadata that fingerprint matching server 106 may receive from or as part of metadata from content distribution system 102, content management system 108, and / or other devices within content modification system 100. For example, the content identifier may be or include content segment identifiers, program identifiers, advertisement identifiers, channel identifiers, and / or other identifiers associated with content presented on content presentation device 104, or be included as part of such identifiers.

[0181] As another example, fingerprint matching server 106 can identify the geographic region (e.g., DMA) of content presentation device 104 and map the identified geographic region to content transmission latency using first mapping data. In practice, content sent from content modification system 102 to one content presentation device in a specific geographic region may have a similar or nearly identical content transmission latency to content sent from content modification system 102 to another content presentation device in the same geographic region.

[0182] The first mapping data can be generated in various ways. For example, the mapping between the above information and content transmission delay can be manually entered. Additionally or alternatively, the fingerprint matching server 106 can collect and store the above information, then determine the content transmission delay between different content distribution systems and content presentation devices over time, and assign each determined content transmission delay to a corresponding device identifier, IP address, content identifier, geographical region, etc. Other examples are also possible. The first mapping data can be created, maintained, and updated over time to record the associations between different types of information (including but not limited to the information described above).

[0183] After the content transmission delay between the content distribution system 102 and the content presentation device 104 has been determined, the fingerprint matching server 106 can use the determined content transmission delay as a basis to select a reference fingerprint dataset corresponding to the determined content transmission delay from multiple reference fingerprint datasets.

[0184] To achieve this, for example, fingerprint matching server 106 may store in memory and / or receive (stored in memory) mapping data (e.g., a table) from other devices within or outside content modification system 100, which maps each of multiple content transmission delays to a corresponding reference fingerprint dataset. This mapping data is referred to herein as "second mapping data". In some examples, first mapping data may include second mapping data. In other examples, first and second mapping data may be stored in separate locations (e.g., separately within the same computing device or remotely to each other in separate computing devices). In either case, any information included in the first mapping data may be mapped to any information in the second mapping data, and vice versa.

[0185] The action of selecting a reference fingerprint dataset corresponding to a determined content transmission delay from multiple reference fingerprint datasets, using at least the determined content transmission delay as a basis, may involve referencing second mapping data to select a reference fingerprint dataset that maps the second mapping data to the determined content transmission delay. The second mapping data may include the selected reference fingerprint dataset, or, once the selected reference fingerprint dataset is identified, the fingerprint matching server 106 may retrieve the selected reference fingerprint dataset from a database or database partition storing the selected reference fingerprint dataset. The database or database partition may be a general database / partition, or it may be allocated to store reference fingerprint datasets associated with a specific content transmission delay.

[0186] The second mapping data can be generated in various ways. For example, the mapping between content delivery delay and reference fingerprint dataset can be manually entered. Additionally or alternatively, the fingerprint matching server 106 can store a list of content delivery delays and then receive reference fingerprint datasets over time, assigning each received reference fingerprint dataset to a corresponding content delivery delay. Other examples are also possible. In some implementations, another device of the content modification system 100, different from the fingerprint matching server 106, can generate and / or store the second mapping data. The action of selecting a reference fingerprint dataset corresponding to a determined content delivery delay, at least based on the determined content delivery delay, can also take other forms.

[0187] In some embodiments, additional information may be stored in memory and / or included as part of the second mapping data. For example, the second mapping data may map each reference fingerprint dataset to a specific channel identifier associated with that reference fingerprint dataset. As another example, the second mapping data may map each of a plurality of device identifiers for a content rendering device to historical content consumption data, demographic data, and / or other data associated with that device identifier. Therefore, the second mapping data can be created, maintained, and updated over time to record associations between different types of information, including but not limited to reference fingerprint datasets, channel identifiers, content delivery latency, device identifiers for content rendering devices, historical content consumption data, and / or demographic data.

[0188] Based on the above discussion, the fingerprint matching server 106 can also use additional information as the basis for selecting a reference fingerprint dataset, such as historical content consumption data, demographic data, and / or other data associated with the device identifier or other identification information of the content presentation device 104. For example, if the device identifier of the content presentation device 104 has been identified, the fingerprint matching server 106 can use the device identifier to look up historical content consumption data, demographic data, and / or other data of the content presentation device 104. Furthermore, when using additional information to select a reference fingerprint dataset, the fingerprint matching server 106 can use the determined content transmission delay and additional information as a basis to identify which reference fingerprint dataset is associated with the determined content transmission delay and additional information, and select the identified reference fingerprint dataset. For example, second mapping data can map a reference fingerprint dataset to a specific content transmission delay and a specific type of content historically consumed by the content presentation device 104. As another example, second mapping data can map a reference fingerprint dataset to a specific content transmission delay and a specific DMA of the content presentation device 104. The fingerprint matching server 106 can use additional information to indicate which reference fingerprint dataset can also be selected in other ways.

[0189] In this way, additional information such as historical content consumption data of the content presentation device 104 can be advantageously used to further refine (e.g., reduce) the number of reference fingerprint datasets considered for determining matches and for channel identification.

[0190] After selecting a reference fingerprint dataset, the fingerprint matching server 106 can send the selected reference fingerprint dataset to the content presentation device 104. The content presentation device 104 can then detect a match between the query fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset received by the content presentation device 104 from the fingerprint matching server 106. The match can be detected in the same or similar manner as described above for cold matching attempts. In response to the detection of a match, the content presentation device 104 can perform an action. The action can take various forms, including but not limited to actions related to other operations that facilitate the content presentation device 104's content modification operations, such as the content presentation device 104 identifying a channel and / or the content presentation device 104 sending an indication of the identified channel to the fingerprint matching server 106.

[0191] As described above, the metadata associated with the query fingerprint data may not identify the channel on which the content presentation device 104 is receiving content. Therefore, the actions performed by the content presentation device 104 may involve identifying a channel. For example, the content presentation device 104 may use at least a portion of the selected reference fingerprint data in a matching dataset to detect that corresponds to a channel, and this action may involve identifying that channel as the channel on which the content presentation device 104 is receiving content. For example, the content presentation device 104 may identify a channel based on channel identifier metadata associated with the reference fingerprint data (e.g., first metadata associated with first fingerprint data that the content presentation device 104 may receive from the content delivery system 102 and / or the fingerprint matching server 106).

[0192] As described above, after the content presentation device 104 identifies a channel on which it is receiving content, it may perform additional actions, such as (i) generating metadata associated with the identified channel on which it is receiving content, (ii) sending an indication of the identified channel and associated metadata (e.g., sending an indication of the identified channel to the fingerprint matching server 106), and / or (iii) performing a content modification operation related to the identified content modification opportunity on the identified channel. Therefore, the actions performed in response to the detection of a match may involve the execution of any one or more of these actions. Other actions are also possible.

[0193] In practice, content presentation device 104 may send and fingerprint matching server 106 may receive a number of queries with query fingerprint data. In such a case, fingerprint matching server 106 may, as appropriate, perform one or more of the operations described above for each of the many queries, such as directing each such query from content presentation device 104 to an appropriate reference fingerprint dataset for content transmission latency between content presentation device 104 and content distribution system.

[0194] Figure 5 This is a flowchart of example method 500. At block 502, method 500 includes determining the content transmission latency between the content distribution system and the content presentation device.

[0195] At block 504, method 500 includes selecting a reference fingerprint dataset corresponding to the determined content transmission delay from a plurality of reference fingerprint datasets, using at least the determined content transmission delay as a basis.

[0196] At block 506, method 500 includes sending a selected reference fingerprint dataset corresponding to the determined content transmission delay to the content rendering device to facilitate the content rendering device in detecting a match between query fingerprint data representing content received by the content rendering device and at least a portion of the reference fingerprint data in the sent reference fingerprint dataset.

[0197] Method 500 and related actions can be performed at least in part by the fingerprint matching server 106. In some implementations, method 500 and related actions can be performed at least in part by another device within the content modification system 100.

[0198] Figure 6 This is a flowchart of another example method 600. At block 602, method 600 includes receiving a reference fingerprint dataset selected from a plurality of reference fingerprint datasets, at least in part based on a reference fingerprint dataset corresponding to the content delivery latency of the content rendering device.

[0199] At block 604, method 600 includes detecting a match between query fingerprint data representing content received by the content rendering device and at least a portion of reference fingerprint data in the received reference fingerprint dataset.

[0200] At block 606, method 600 includes performing an action by the content rendering device in response to detecting a match.

[0201] Method 600 and related actions may be performed at least in part by the content presentation device 104. In some embodiments, method 600 and related actions may be performed at least in part by another device within the content modification system 100.

[0202] Figure 7 This is a flowchart of another example method 700. At block 702, method 700 includes determining the content transmission delay between the content distribution system and the content presentation device.

[0203] At block 704, method 700 includes selecting a reference fingerprint dataset corresponding to the determined content transmission delay from a plurality of reference fingerprint datasets, using at least the determined content transmission delay as a basis.

[0204] At block 706, method 700 includes sending a selected reference fingerprint dataset corresponding to the determined content transmission delay to the content rendering device.

[0205] At block 708, method 700 includes receiving the transmitted reference fingerprint dataset by the content rendering device.

[0206] At block 710, method 700 includes detecting a match between query fingerprint data representing content received by the content presentation device and at least a portion of the received reference fingerprint data.

[0207] At block 712, method 700 includes performing an action by the content rendering device in response to detecting a match.

[0208] Blocks 702 to 706 of method 700 and related actions can be performed at least partially by fingerprint matching server 106. In some embodiments, blocks 702 to 706 of method 700 and related actions can be performed at least partially by another device within content modification system 100.

[0209] As described above, the implementation of the disclosed methods and systems provides technical improvements specific to computer networks and computing systems (e.g., content modification systems such as content modification system 100).

[0210] Systems configured to generate and deliver content (e.g., target content) to end users may present various technical challenges, such as inefficiency and computational complexity. These challenges are particularly pronounced in content modification systems, where numerous content distribution systems send large volumes of video content to numerous content presentation devices over extended periods when modifications are required to many content segments received and presented by these content presentation devices. The aforementioned operations and their various implementations can be integrated into these and other content modification system environments to address these and other technical challenges, in whole or in part.

[0211] For example, as described above, embodiments of this disclosure improve the overall content modification process by enabling the content presentation device to more quickly identify the channel on which it is receiving content, thereby allowing the content modification system as a whole to more quickly identify content modification opportunities and act accordingly. In practice, operations for modifying content and providing content to end users typically occur within a very short time (e.g., a few seconds or less). Therefore, the disclosed operations help to make various aspects of content modification more efficient, namely fingerprint matching, channel identification, and, importantly, actions performed as a result of fingerprint matching and channel identification, such as identifying upcoming content modification opportunities and / or preparing the content presentation device to perform content modification operations related to those opportunities. This helps ensure that these operations are reliably performed within the expected timeframe, including in large content modification systems.

[0212] As another example, as described above, because embodiments of this disclosure can use content transmission latency (and possibly additional information) to limit which reference fingerprint datasets are considered, these embodiments can reduce the amount of storage and / or processing power required by the content presentation device to store reference fingerprint datasets, perform fingerprint matching, and identify channels. By making the fingerprint matching process and subsequent actions less computationally expensive in this way, various aspects of content modification can be made more efficient and reliable.

[0213] These and other improvements have been described in detail herein. Although the operations described herein are for illustrative purposes, other improvements may be provided for the implementation of the disclosed methods and systems.

[0214] IV. Example Variation

[0215] Although the examples and characteristics of specific entities and operations described above have been described, in practice, numerous instances of these entities and numerous instances of these operations may be executed concurrently or at a large scale. In practice, content modification system 100 may include numerous content distribution systems (each potentially sending content on numerous channels) and numerous content presentation devices, where some or all of the desired operations performed on a routine and repetitive basis are related to some or all of these entities.

[0216] Furthermore, while some operations described in this disclosure have been described as being performed by a specific entity, these operations can be performed by any entity, such as other entities described in this disclosure. Moreover, although operations have been described in a specific order and / or in conjunction with example time language, it is not necessary to perform the operations in the described order, nor is it necessary to perform the operations according to any specific time constraints. However, in some cases, it may be desirable to perform one or more operations in the stated order, in another order, and / or in a manner that at least some of the operations are performed concurrently / simultaneously. Similarly, in some cases, it may be necessary to perform one or more operations according to one or more of the enumerated time constraints or other timing constraints. Furthermore, each of the described operations may be performed in response to the execution of one or more operations of the other described operations. Furthermore, not all operations need to be performed to achieve one or more benefits provided by this disclosure, and therefore not all operations are necessary.

[0217] Although certain variations have been described in conjunction with one or more examples of this disclosure, these variations may also be applied to some or all of other examples of this disclosure, and thus aspects of this disclosure may be combined and / or arranged in a variety of ways. The examples described in this disclosure were chosen at least in part because they help to explain the practical application of the various features described.

[0218] Furthermore, while examples of alternatives have been described in this disclosure, variations and arrangements of these examples will be readily apparent to those skilled in the art. Other changes, substitutions, and / or alterations are also possible without departing from the broader aspects of the invention as set forth in the appended claims.

Claims

1. A method for performing an action based on a feature of transmission delay, comprising: Receive query fingerprint data representing the content received by the content presentation device; Determine the content transmission delay between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset corresponding to the determined content transmission delay is selected from multiple reference fingerprint datasets. This selection of reference fingerprint datasets corresponding to the determined content transmission delay from multiple reference fingerprint datasets includes: Access mapping data, which maps each of a plurality of content delivery delays to a corresponding reference fingerprint dataset of a plurality of reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission delay; Detect a match between the queried fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset; and An action is performed in response to the detection of the match.

2. The method according to claim 1, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

3. The method according to claim 1, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: The device identifier that identifies the content presentation device; and Map the identified device identifier to the content transmission delay.

4. The method according to claim 1, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the IP address of the content presentation device; and Map the identified IP address to the content transmission delay.

5. The method according to claim 1, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the content presented on the content presentation device; and Map the identified content to the content transmission delay.

6. The method according to claim 1, wherein, The actions performed include actions that facilitate the content presentation device to perform content modification operations.

7. The method of claim 6, wherein at least a portion of the selected reference fingerprint dataset corresponds to a channel, and wherein, The action includes identifying the channel as the channel on which the content presentation device is receiving content.

8. A non-transitory computer-readable storage medium having program instructions stored thereon, the program instructions causing a set of operations to be performed when executed by a processor, the operations including: Receive query fingerprint data representing the content received by the content presentation device; Determine the content transmission delay between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset corresponding to the determined content transmission delay is selected from multiple reference fingerprint datasets. This selection of reference fingerprint datasets corresponding to the determined content transmission delay from multiple reference fingerprint datasets includes: Access mapping data, which maps each of a plurality of content delivery delays to a corresponding reference fingerprint dataset of a plurality of reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission delay; Detect a match between the queried fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset; and An action is performed in response to the detection of the match.

9. The non-transitory computer-readable storage medium according to claim 8, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

10. The non-transitory computer-readable storage medium according to claim 8, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: The device identifier that identifies the content presentation device; and Map the identified device identifier to the content transmission delay.

11. The non-transitory computer-readable storage medium according to claim 8, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the IP address of the content presentation device; and Map the identified IP address to the content transmission delay.

12. The non-transitory computer-readable storage medium of claim 8, wherein determining the content transmission delay between the content distribution system and the content presentation device comprises: Identify the content presented on the content presentation device; and Map the identified content to the content transmission delay.

13. The non-transitory computer-readable storage medium according to claim 8, wherein, Performing the action includes performing actions that facilitate the content presentation device to perform content modification operations.

14. The non-transitory computer-readable storage medium according to claim 13, wherein, At least a portion of the selected reference fingerprint dataset corresponds to a channel, and the action includes identifying the channel as the channel on which the content presentation device is receiving content.

15. A computing system, comprising: processor; and A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by a processor, cause the execution of a set of operations, the operations including: Receive query fingerprint data representing the content received by the content presentation device; Determine the content transmission delay between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset corresponding to the determined content transmission delay is selected from multiple reference fingerprint datasets. This selection of reference fingerprint datasets corresponding to the determined content transmission delay from multiple reference fingerprint datasets includes: Access mapping data, which maps each of a plurality of content delivery delays to a corresponding reference fingerprint dataset of a plurality of reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission delay; Detect a match between the queried fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset; and An action is performed in response to the detection of the match.

16. The system according to claim 15, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

17. The system according to claim 15, wherein, The actions performed include actions that facilitate the content presentation device to perform content modification operations.

18. The system of claim 17, wherein at least a portion of the selected reference fingerprint dataset corresponds to a channel, and wherein, Performing the action includes identifying the channel as a channel on which the content presentation device is receiving content.

19. The system according to claim 15, wherein, The content distribution system is a television broadcasting station.

20. The system according to claim 15, wherein, The computing system is a fingerprint matching server for the content modification system.

21. A method for performing content modification based on characteristics based on transmission delay, comprising: Receive query fingerprint data representing the content received by the content presentation device; Determine the content transmission delay between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset is selected from multiple reference fingerprint datasets to map the mapping data to the determined content transmission delay. This reference fingerprint dataset, which uses at least the determined content transmission delay as a basis, includes the following: Access the mapping data, which maps each of the multiple content delivery delays to a corresponding reference fingerprint dataset of multiple reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission delay; Detect a match between the query fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset, wherein at least a portion of the reference fingerprint data in the selected reference fingerprint dataset corresponds to a channel; Based on the matching, the channel is identified as the channel on which the content presentation device is receiving content; and In response to the detection of the match, the content presentation device performs a content modification operation related to the content modification opportunity identified on the identified channel.

22. The method according to claim 21, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

23. The method of claim 21, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: The device identifier that identifies the content presentation device; and Map the identified device identifier to the content transmission delay.

24. The method of claim 21, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the IP address of the content presentation device; and Map the identified IP address to the content transmission delay.

25. The method of claim 21, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the content presented on the content presentation device; and Map the identified content to the content transmission delay.

26. The method of claim 21, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the geographical region of the content presentation device; and Map the identified geographic regions to the content transmission latency.

27. The method according to claim 21, wherein, The content presentation device performing a content modification operation related to a content modification opportunity identified on the identified channel includes: causing the content presentation device to output supplementary content to replace the modifiable content segment received by the content presentation device on the identified channel.

28. The method according to claim 21, wherein, The content presentation device performing a content modification operation related to a content modification opportunity identified on the identified channel includes: causing the content presentation device to overlay supplementary content onto a modifiable content segment received by the content presentation device on the identified channel.

29. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by a processor, cause a set of operations to be performed, the operations including: Receive query fingerprint data representing the content received by the content presentation device; Determine the content transmission delay between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset is selected from multiple reference fingerprint datasets to map the mapping data to the determined content transmission delay. This reference fingerprint dataset, which uses at least the determined content transmission delay as a basis, includes the following: Access the mapping data, which maps each of the multiple content delivery delays to a corresponding reference fingerprint dataset of multiple reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission delay; Detect a match between the query fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset, wherein at least a portion of the reference fingerprint data in the selected reference fingerprint dataset corresponds to a channel; Based on the matching, the channel is identified as the channel on which the content presentation device is receiving content; and In response to the detection of the match, the content presentation device performs a content modification operation related to the content modification opportunity identified on the identified channel.

30. The non-transitory computer-readable storage medium according to claim 29, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

31. The non-transitory computer-readable storage medium of claim 29, wherein determining the content transmission delay between the content distribution system and the content presentation device comprises: A device identifier that identifies the device presenting the content; and Map the identified device identifier to the content transmission delay.

32. The non-transitory computer-readable storage medium of claim 29, wherein determining the content transmission delay between the content distribution system and the content presentation device comprises: Identify the IP address of the content presentation device; and Map the identified IP address to the content transmission delay.

33. The non-transitory computer-readable storage medium according to claim 29, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the content presented on the content presentation device; and Map the identified content to the content transmission delay.

34. The non-transitory computer-readable storage medium according to claim 29, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the geographical region of the content presentation device; and Map the identified geographic regions to the content transmission latency.

35. The non-transitory computer-readable storage medium according to claim 29, wherein, The content presentation device performing a content modification operation related to a content modification opportunity identified on the identified channel includes: causing the content presentation device to output supplementary content to replace the modifiable content segment received by the content presentation device on the identified channel.

36. The non-transitory computer-readable storage medium according to claim 29, wherein, The content presentation device performing a content modification operation related to a content modification opportunity identified on the identified channel includes: causing the content presentation device to overlay supplementary content onto a modifiable content segment received by the content presentation device on the identified channel.

37. A computing system, comprising: processor; and A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by a processor, cause a set of operations to be performed, the operations including: Receive query fingerprint data representing the content received by the content presentation device; Determine the content transmission delay between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset is selected from multiple reference fingerprint datasets to map the mapping data to the determined content transmission delay. This reference fingerprint dataset, which uses at least the determined content transmission delay as a basis, includes the following: Access the mapping data, which maps each of the multiple content delivery delays to a corresponding reference fingerprint dataset of multiple reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission delay; Detect a match between the query fingerprint data and at least a portion of the reference fingerprint data in the selected reference fingerprint dataset, wherein at least a portion of the reference fingerprint data in the selected reference fingerprint dataset corresponds to a channel; Based on the matching, the channel is identified as the channel on which the content presentation device is receiving content; and In response to the detection of the match, the content presentation device performs a content modification operation related to the content modification opportunity identified on the identified channel.

38. The computing system according to claim 37, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

39. The computing system according to claim 37, wherein, The computing system is a fingerprint matching server.

40. The computing system according to claim 37, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Identify the geographical region of the content presentation device; and Map the identified geographic regions to the content transmission latency.

41. A method for performing fingerprint matching based on features based on transmission delay, comprising: Determine the content transmission latency between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset corresponding to the determined content transmission delay is selected from multiple reference fingerprint datasets. The selection of the reference fingerprint dataset corresponding to the determined content transmission delay using at least the determined content transmission delay as a basis includes: Access mapping data, which maps each of a plurality of content delivery delays to a corresponding reference fingerprint dataset of a plurality of reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission latency; and A selected reference fingerprint dataset corresponding to a determined content transmission delay is sent to the content presentation device to facilitate the content presentation device in detecting a match between query fingerprint data representing content received by the content presentation device and at least a portion of the reference fingerprint data in the sent reference fingerprint dataset.

42. The method of claim 41, wherein determining the content transmission delay between the content distribution system and the content presentation device includes determining the content transmission delay as the difference between the sending timestamp of the content sent by the content distribution system and the receiving timestamp of the same content received by the content presentation device.

43. The method of claim 41, wherein determining the content transmission delay between the content distribution system and the content presentation device comprises: Determine the device identifier of the device that presents the content; Access the second mapping data, which maps each of the multiple device identifiers of the content rendering device to a corresponding content delivery delay; and The second mapping data is used as a basis to select the content transmission delay for mapping data to the determined device identifier.

44. The method according to claim 41, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Determine the Internet Protocol (IP) address of the content presentation device; Access the second mapping data, which maps each of the multiple IP addresses to a corresponding content transmission delay; and The second mapping data is used as a basis to select the content transmission delay for mapping data to the determined IP address.

45. The method according to claim 41, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Determine the geographical region of the content presentation device; Access the second mapping data, which maps each geographic region in a plurality of geographic regions to a corresponding content delivery latency; and The second mapping data is used as a basis to select the content transmission delay for mapping data to the determined geographic area.

46. ​​The method of claim 41, further comprising: Based on the historical content consumption data of the content presentation device, the types of content historically received by the content presentation device are determined. The mapping data further maps each of the plurality of reference fingerprint datasets to content of the corresponding type historically received by the content presentation device. The selection of a reference fingerprint dataset corresponding to the determined content transmission delay, at least using the determined content transmission delay as a basis, also includes selecting a reference fingerprint dataset that uses mapping data as a basis to map the mapping data to the determined content transmission delay and the determined types of content historically received by the content presentation device.

47. The method according to claim 41, wherein, The content presentation device is configured to perform a set of operations, including: Receive the transmitted reference fingerprint dataset; Detecting a match between the queried fingerprint data and at least a portion of the reference fingerprint data in the received reference fingerprint dataset; and An action is performed in response to the detection of the match.

48. The method according to claim 47, wherein, The actions performed include actions that facilitate the content presentation device to perform content modification operations.

49. The method of claim 48, wherein at least a portion of the reference fingerprint data in the selected reference fingerprint dataset corresponds to a channel, and wherein, The action includes identifying the channel as a channel on which the content presentation device is presenting or receiving content.

50. The method of claim 41, wherein, The content distribution system includes television broadcasting stations. The content presentation device includes a television set, and The content received by the content presentation device includes video content.

51. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by a processor, cause a set of operations to be performed, the operations including: Determine the content transmission latency between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset corresponding to the determined content transmission delay is selected from multiple reference fingerprint datasets. The selection of the reference fingerprint dataset corresponding to the determined content transmission delay using at least the determined content transmission delay as a basis includes: Access mapping data, which maps each of a plurality of content delivery delays to a corresponding reference fingerprint dataset of a plurality of reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission latency; and A selected reference fingerprint dataset corresponding to a determined content transmission delay is sent to the content presentation device to facilitate the content presentation device in detecting a match between query fingerprint data representing content received by the content presentation device and at least a portion of the reference fingerprint data in the sent reference fingerprint dataset.

52. The non-transitory computer-readable storage medium according to claim 51, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Determine the device identifier of the device that presents the content; Access the second mapping data, which maps each of the multiple device identifiers of the content rendering device to a corresponding content delivery delay; and The second mapping data is used as a basis to select the content transmission delay for mapping data to the determined device identifier.

53. The non-transitory computer-readable storage medium according to claim 51, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Determine the Internet Protocol (IP) address of the content presentation device; Access the second mapping data, which maps each of the multiple IP addresses to a corresponding content transmission delay; and The second mapping data is used as a basis to select the content transmission delay for mapping data to the determined IP address.

54. The non-transitory computer-readable storage medium according to claim 51, wherein, Determining the content transmission delay between the content distribution system and the content presentation device includes: Determine the geographical region of the content presentation device; Access the second mapping data, which maps each geographic region in a plurality of geographic regions to a corresponding content delivery latency; and The second mapping data is used as a basis to select the content transmission delay for mapping data to the determined geographic area.

55. The non-transitory computer-readable storage medium of claim 51, wherein the set of operations further comprises: Based on the historical content consumption data of the content presentation device, the types of content historically received by the content presentation device are determined. The mapping data further maps each of the plurality of reference fingerprint datasets to content of the corresponding type historically received by the content presentation device. The selection of a reference fingerprint dataset corresponding to the determined content transmission delay, at least using the determined content transmission delay as a basis, also includes selecting a reference fingerprint dataset that uses mapping data as a basis to map the mapping data to the determined content transmission delay and the determined types of content historically received by the content presentation device.

56. The non-transitory computer-readable storage medium of claim 51, wherein the content rendering device is configured to perform a set of operations, the operations including: Receive the transmitted reference fingerprint dataset; Detect a match between the queried fingerprint data and at least a portion of the reference fingerprint data in the received reference fingerprint dataset; as well as An action is performed in response to the detection of the match.

57. The non-transitory computer-readable storage medium according to claim 56, wherein, The actions performed include actions that facilitate the content presentation device to perform content modification operations.

58. The non-transitory computer-readable storage medium according to claim 57, wherein, At least a portion of the reference fingerprint data in the selected reference fingerprint dataset corresponds to a channel, and the action includes identifying the channel as the channel on which the content presentation device is presenting or receiving content.

59. The non-transitory computer-readable storage medium according to claim 51, wherein, The content distribution system includes television broadcasting stations. The content presentation device includes a television set, and The content received by the content presentation device includes video content.

60. A computing system, comprising: processor; and A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by a processor, cause a set of operations to be performed, the operations including: Determine the content transmission latency between the content distribution system and the content presentation device; Using at least the determined content transmission delay as a basis, a reference fingerprint dataset corresponding to the determined content transmission delay is selected from multiple reference fingerprint datasets. The selection of the reference fingerprint dataset corresponding to the determined content transmission delay using at least the determined content transmission delay as a basis includes: Access mapping data, which maps each of a plurality of content delivery delays to a corresponding reference fingerprint dataset of a plurality of reference fingerprint datasets, and The mapping data is used as a basis to select a reference fingerprint dataset to which the mapping data is mapped to the determined content transmission latency; and A selected reference fingerprint dataset corresponding to a determined content transmission delay is sent to the content presentation device to facilitate the content presentation device in detecting a match between query fingerprint data representing content received by the content presentation device and at least a portion of the reference fingerprint data in the sent reference fingerprint dataset.

Images